Neural Networks: How Optimization Actually Converges

Introduction: the ball in a bowl that does not exist

The popular picture of learning shows a ball rolling down a bowl. Height is loss, position is the parameter vector, the gradient is the slope, and the ball reaches the bottom. This picture is locally useful and globally misleading.

Modern networks may have millions or billions of parameters. Their loss functions contain flat directions, symmetries, saddles, connected valleys and changing curvature. Training usually sees only mini-batches. Gradients are noisy. Numbers have finite precision. Computation is distributed. Yet optimization works.

The real question is not simply how to move downhill. It is why local, noisy and incomplete updates find useful functions in such enormous spaces.

What exactly converges?

Possible notions include: training loss, gradient norm, parameters, predictions, update distribution, validation performance, a low-loss region. They are not equivalent.

Weights may keep moving while predictions barely change. Training loss may improve while validation worsens. A network may converge to a valley rather than one point.

A parameterized function

Write the network as f_θ(x). The empirical loss is the average of ℓ(f_θ(xᵢ), yᵢ) over all examples. Training seeks low empirical loss, but the objective is nonconvex and the dataset is only a sample.

The objective is nonconvex. The dataset is only a sample. Low training loss does not guarantee low future error. Many parameter vectors may represent nearly the same function.

Backpropagation and the gradient

Backpropagation applies the chain rule efficiently. It computes ∇_θ L. It is not the optimizer. SGD, AdamW, SAM or Muon decide how to turn that gradient into an update.

Gradient descent on a quadratic

For L(θ) = ½ θᵀ H θ, the update is θ_{t+1} = (I − η H) θ_t. In an eigenvector direction: z_{t+1,i} = (1 − η λᵢ) z_{t,i}. Convergence requires 0 < η < 2/λ_max. The largest curvature limits the learning rate.

Curvature and the Hessian

The Hessian H = ∇²L describes local gradient variation. The directional curvature is v^T H v. Positive eigenvalues curve upward. Negative eigenvalues indicate saddle directions. Near-zero eigenvalues indicate flat directions.

The full Hessian is usually too large to form. Practical analysis uses Hessian-vector products, power iteration or Lanczos. The local Hessian does not describe the full global geometry.

Ill-conditioned valleys

In a narrow valley, the gradient points mainly across the valley. Plain gradient descent zigzags. The condition number κ = λ_max/λ_min measures anisotropy for a positive quadratic. One global step size must obey the stiffest direction.

Learning rate

A small learning rate is stable but slow. A moderate one is efficient and may oscillate. A large one can diverge. Curvature changes during training, so practical learning rates use schedules.

Momentum and local spirals

Polyak momentum uses v_{t+1} = μ v_t + g_t and θ_{t+1} = θ_t − η v_{t+1}. For a quadratic, each eigendirection follows a second-order recurrence. Complex roots generate damped oscillation.

A 2D projection may therefore spiral toward a fixed point. This is one real origin of an optimization spiral. Momentum may accelerate persistent directions but can destabilize badly chosen settings.

Nesterov momentum

Nesterov evaluates the gradient at an extrapolated point. It is a precise mathematical correction, not literal future prediction. Its convex acceleration theory is important, while deep-network behavior depends on noise and schedules.

Stochastic gradient noise

A mini-batch gradient can be written g_B = g + ξ. The noise covariance depends on data, model, batch and position. Noise may help exploration, act as implicit regularization or create instability. SGD is a stochastic dynamical system.

Batch size and critical scale

Small batches create noisy updates and more steps per epoch. Large batches improve parallelism and gradient accuracy. Beyond a critical scale, larger batches may give diminishing returns. Linear learning-rate scaling works only in a limited regime. Warmup is often required.

Adam and adaptive preconditioning

Adam maintains first and second moments. The update divides by √v̂_t + ε. Different coordinates receive different effective step sizes. Adam is a diagonal adaptive preconditioner with momentum. It is widely used for transformers and language models.

AdamW

AdamW decouples weight decay from the adaptive gradient update. This changes the optimization geometry and generally makes regularization easier to tune. It is widely used for transformers and language models.

Adam convergence caveats

Adam is empirically successful but does not possess a universal convergence guarantee. Counterexamples motivated AMSGrad and further theory. Empirical success and mathematical guarantee must remain separate.

Deep loss landscapes

A 2D landscape is usually L(θ₀ + α d₁ + β d₂). The picture depends on the chosen directions and normalization. It is a slice, not the complete geometry.

Saddles, plateaus and symmetries

High-dimensional critical points often have saddle directions. Neuron permutations and layer rescalings create families of equivalent parameters. Minima are frequently manifolds or connected regions rather than isolated points.

Overparameterization

Overparameterization creates many interpolating solutions. It can open low-loss paths and simplify optimization. The modern question is often which interpolating solution the optimizer selects.

Why zero loss can be easy to reach

In the interpolation regime, the network fits all training examples. This does not automatically mean pure memorization. The modern question is which interpolating solution the optimizer selects.

Implicit bias

Optimization chooses among many solutions even without explicit regularization. Examples include minimum-norm linear regression and maximum-margin directions for separable logistic classification. The optimizer is therefore part of the model-selection mechanism.

Margin and directional convergence

For separable logistic regression, weight norms may diverge while normalized direction converges. Parameter convergence is not always the right object. Function or direction convergence may be more meaningful.

Flatness and sharpness

Flat regions may tolerate parameter perturbations. Sharpness can be measured by Hessian eigenvalues or worst-case neighborhood loss. But sharpness depends on coordinates, scaling and metric.

Reparameterization caveat

ReLU layer rescaling can change Hessian sharpness without changing the represented function. Sharpness measures must specify normalization and geometry.

SAM

SAM optimizes min_θ max_{‖ε‖≤ρ} L(θ+ε). It seeks neighborhoods with uniformly low loss. It can improve generalization but costs additional computation and is not universally superior.

Edge of stability

Classical quadratic theory suggests η λ_max < 2. In neural-network training, λ_max often rises near 2/η. Short-term loss oscillates while long-term loss decreases. This edge-of-stability regime shows that the network and curvature coevolve.

Learning-rate schedules

Warmup increases the rate gradually. Cosine decay lowers it smoothly. Restarts periodically raise it. Schedules affect stability, noise and implicit bias, not only speed.

Gradient clipping and mixed precision

Norm clipping limits exploding updates. Mixed precision uses FP16 or BF16 with higher-precision accumulation. Loss scaling prevents underflow in FP16. Numerical representation is part of practical optimization.

Distributed optimization

Large models distribute gradients, parameters and optimizer states. Communication, synchronization, batch size and fault tolerance matter. Adam stores two moments per parameter. Sharding methods distribute this memory across devices. Optimization becomes a systems problem.

Muon and matrix-aware optimizers

Muon treats weight matrices as structured objects. A practical version computes matrix momentum, approximately orthogonalizes it with Newton–Schulz iterations, rescales the update, and often adds weight decay.

Recent studies report promising training efficiency on some transformers and language models. Theory is still developing. Muon should be presented as an emerging direction, not a settled replacement for AdamW.

Honest visualization

A toy quadratic can be shown exactly. A neural-network surface is only a slice. A training path is only a projection. High-dimensional statistics should accompany visual pictures.

A spiral visible at level 1 may be exact. At level 3, it may be a projection artifact. Each figure must indicate its level, directions, normalization and scale.

Interactive laboratory

The following modules allow exploring the dynamics described in this article. All data is synthetic and reproducible by seed. No computation represents a real training run.

Where is the spiral?

A local second-order system can have complex eigenvalues and spiral toward a fixed point. A projected deep-learning trajectory may also look spiral. But deep learning is not universally spiral descent.

The robust shared ideas are iteration, memory, oscillation and convergence. Rigor means showing the spiral when it is mathematically present and refusing to impose it when it is not.

Conclusion

Neural-network optimization combines local gradients, momentum, noise, preconditioning, overparameterization, schedules, implicit bias, and numerical and distributed systems.

Training may converge to a function, direction, valley, distribution or stable regime. The spiral is only one possible form of this dynamics.

The essential question is not merely where the minimum lies. It is which solution the dynamics select and why that solution generalizes.