None
EN
Why Momentum Really Works
['Goh', 'Uc Davis']
Distill
⋆ \star ⋆ + + + − - − = = = α \alpha α λ \lambda λ β \beta β R R R α = \alpha= α = β = \beta= β = β = 0 \beta = 0 β = 0 β = 1 \beta=1 β = 1 α = 1 / λ i \alpha = 1/\lambda_i α = 1 / λ i m o d e l \text{model} model 0 p 1 0 p_1 0 p 1 0 p ¯ 1 0 \bar{p}_1 0 p ¯ 1 2 β 2\sqrt{\beta} 2 √ β λ i \lambda_i λ i λ i = 0 \lambda_i = 0 λ i = 0 α > 1 / λ i \alpha > 1/\lambda_i α > 1 / λ i max { ∣ σ 1 ∣ , ∣ σ 2 ∣ } > 1 \max\{|\sigma_1|,|\sigma_2|\} > 1 max { ∣ σ 1 ∣ , ∣ σ 2 ∣ } > 1 x i k − x i ∗ x_i^k - x_i^* x i k − x i ∗ ξ i \xi_i ξ i β = ( 1 − α λ i ) 2 \beta = (1 - \sqrt{\alpha \lambda_i})^2 β = ( 1 − √ α λ i ) 2 The overall convergence rate is determined by the slowest error component, which…