Neural Networks: How Optimization Actually Converges

The popular picture of learning shows a ball rolling down a bowl. Height is loss, position is the parameter vector, the gradient is the slope, and the ball reaches the bottom. This picture is locally useful and globally misleading.

Modern networks may have millions or billions of parameters. Their loss functions contain flat directions, symmetries, saddles, connected valleys and changing curvature. Training usually sees only mini-batches. Gradients are noisy. Numbers have finite precision. Computation is distributed. Yet optimization works.

The real question is not simply how to move downhill. It is why local, noisy and incomplete updates find useful functions in such enormous spaces.

Possible notions include: training loss, gradient norm, parameters, predictions, update distribution, validation performance, a low-loss region. They are not equivalent.

Weights may keep moving while predictions barely change. Training loss may improve while validation worsens. A network may converge to a valley rather than one point.

Write the network as f_θ(x). The empirical loss is the average of ℓ(f_θ($x_{i}$), $y_{i}$) over all examples. Training seeks low empirical loss, but the objective is nonconvex and the dataset is only a sample.

The objective is nonconvex. The dataset is only a sample. Low training loss does not guarantee low future error. Many parameter vectors may represent nearly the same function.

Backpropagation applies the chain rule efficiently. It computes ∇_θ L. It is not the optimizer. SGD, AdamW, SAM or Muon decide how to turn that gradient into an update.

For L(θ) = ½ $θ^{T}$ H θ, the update is $\theta_{t+1}=(I-\eta H)\theta_t$. In an eigenvector direction: $z_{t+1,i}=(1-\eta\lambda_i)z_{t,i}$. Convergence requires $0<\eta<\frac2{\lambda_{\max}}$. The largest curvature limits the learning rate.

[[figure:QuadraticValley]]

The Hessian H = ∇²L describes local gradient variation. The directional curvature is v^T H v. Positive eigenvalues curve upward. Negative eigenvalues indicate saddle directions. Near-zero eigenvalues indicate flat directions.

The full Hessian is usually too large to form. Practical analysis uses Hessian-vector products, power iteration or Lanczos. The local Hessian does not describe the full global geometry.

[[figure:HessianSpectrum]]

In a narrow valley, the gradient points mainly across the valley. Plain gradient descent zigzags. The condition number $\kappa=\frac{\lambda_{\max}}{\lambda_{\min}}$ measures anisotropy for a positive quadratic. One global step size must obey the stiffest direction.

[[figure:ConditioningZigzag]]

A small learning rate is stable but slow. A moderate one is efficient and may oscillate. A large one can diverge. Curvature changes during training, so practical learning rates use schedules.

[[figure:LearningRateRegimes]]

Polyak momentum uses $v_{t+1}$ = μ $v_{t}$ + $g_{t}$ and $θ_{t+1}$ = $θ_{t}$ − η $v_{t+1}$. For a quadratic, each eigendirection follows a second-order recurrence. Complex roots generate damped oscillation.

A 2D projection may therefore spiral toward a fixed point. This is one real origin of an optimization spiral. Momentum may accelerate persistent directions but can destabilize badly chosen settings.

[[figure:MomentumSpiral]]

Nesterov evaluates the gradient at an extrapolated point. It is a precise mathematical correction, not literal future prediction. Its convex acceleration theory is important, while deep-network behavior depends on noise and schedules.

[[figure:NesterovComparison]]

A mini-batch gradient can be written $g_B=g+\xi$. The noise covariance depends on data, model, batch and position. Noise may help exploration, act as implicit regularization or create instability. SGD is a stochastic dynamical system.

[[figure:SGDNoise]]

Small batches create noisy updates and more steps per epoch. Large batches improve parallelism and gradient accuracy. Beyond a critical scale, larger batches may give diminishing returns. Linear learning-rate scaling works only in a limited regime. Warmup is often required.

[[figure:BatchSizeTable]]

Adam maintains first and second moments. The update divides by $\sqrt{\hat v_t}+\varepsilon$. Different coordinates receive different effective step sizes. Adam is a diagonal adaptive preconditioner with momentum. It is widely used for transformers and language models. ^[1]

[[figure:AdamDiagonal]]

AdamW decouples weight decay from the adaptive gradient update. This changes the optimization geometry and generally makes regularization easier to tune. It is widely used for transformers and language models. ^[2]

Adam is empirically successful but does not possess a universal convergence guarantee. Counterexamples motivated AMSGrad and further theory. Empirical success and mathematical guarantee must remain separate.

A 2D landscape is usually L($θ_{0}$ + α $d_{1}$ + β $d_{2}$). The picture depends on the chosen directions and normalization. It is a slice, not the complete geometry.

[[figure:LossLandscapeSlice]]

High-dimensional critical points often have saddle directions. Neuron permutations and layer rescalings create families of equivalent parameters. Minima are frequently manifolds or connected regions rather than isolated points.

[[figure:SaddlePoint]]

Overparameterization creates many interpolating solutions. It can open low-loss paths and simplify optimization. The modern question is often which interpolating solution the optimizer selects.

In the interpolation regime, the network fits all training examples. This does not automatically mean pure memorization. The modern question is which interpolating solution the optimizer selects.

Optimization chooses among many solutions even without explicit regularization. Examples include minimum-norm linear regression and maximum-margin directions for separable logistic classification. The optimizer is therefore part of the model-selection mechanism.

[[figure:ImplicitBiasNorm]]

For separable logistic regression, weight norms may diverge while normalized direction converges. Parameter convergence is not always the right object. Function or direction convergence may be more meaningful.

[[figure:MarginConvergence]]

Flat regions may tolerate parameter perturbations. Sharpness can be measured by Hessian eigenvalues or worst-case neighborhood loss. But sharpness depends on coordinates, scaling and metric.

ReLU layer rescaling can change Hessian sharpness without changing the represented function. Sharpness measures must specify normalization and geometry.

SAM optimizes min_θ $max_{‖ε‖≤ρ}$ L(θ+ε). It seeks neighborhoods with uniformly low loss. It can improve generalization but costs additional computation and is not universally superior.

Classical quadratic theory suggests $\eta\lambda_{\max}<2$. In neural-network training, $\lambda_{\max}$ often rises near 2/η. Short-term loss oscillates while long-term loss decreases. This edge-of-stability regime shows that the network and curvature coevolve.

[[figure:EdgeOfStability]]

Warmup increases the rate gradually. Cosine decay lowers it smoothly. Restarts periodically raise it. Schedules affect stability, noise and implicit bias, not only speed.

[[figure:CosineSchedule]]

Norm clipping limits exploding updates. Mixed precision uses FP16 or BF16 with higher-precision accumulation. Loss scaling prevents underflow in FP16. Numerical representation is part of practical optimization.

Large models distribute gradients, parameters and optimizer states. Communication, synchronization, batch size and fault tolerance matter. Adam stores two moments per parameter. Sharding methods distribute this memory across devices. Optimization becomes a systems problem.

[[figure:DistributedPipeline]]

Muon treats weight matrices as structured objects. A practical version computes matrix momentum, approximately orthogonalizes it with Newton–Schulz iterations, rescales the update, and often adds weight decay.

Recent studies report promising training efficiency on some transformers and language models. Theory is still developing. Muon should be presented as an emerging direction, not a settled replacement for AdamW.

A toy quadratic can be shown exactly. A neural-network surface is only a slice. A training path is only a projection. High-dimensional statistics should accompany visual pictures.

A spiral visible at level 1 may be exact. At level 3, it may be a projection artifact. Each figure must indicate its level, directions, normalization and scale.

The following modules allow exploring the dynamics described in this article. All data is synthetic and reproducible by seed. No computation represents a real training run.

[[figure:OptimizerComparison]]

[[figure:SmallClassifier]]

A local second-order system can have complex eigenvalues and spiral toward a fixed point. A projected deep-learning trajectory may also look spiral. But deep learning is not universally spiral descent.

The robust shared ideas are iteration, memory, oscillation and convergence. Rigor means showing the spiral when it is mathematically present and refusing to impose it when it is not.

Neural-network optimization combines local gradients, momentum, noise, preconditioning, overparameterization, schedules, implicit bias, and numerical and distributed systems.

Training may converge to a function, direction, valley, distribution or stable regime. The spiral is only one possible form of this dynamics.

The essential question is not merely where the minimum lies. It is which solution the dynamics select and why that solution generalizes.