SoftServe: A Scalable Quasi-Newton Method for Deep Learning
arXiv cs.AI · 2 Oct 2026, 03:58 AEST
Quasi-Newton (QN) methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes.
Key facts
- Quasi-Newton (QN) methods have long been among the most effective methods for large-scale unconstrained convex optimization.
- Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes.
- We introduce SoftServe, a family of QN methods designed to overcome these obstacles without line searches or ad hoc curvature corrections.
- SoftServe derives positivedefinite curvature estimates from the variational objective of Berglund et al. (2025), even in the presence of negative curvature.
- We develop diagonal and Kroneckerfactored variants that preserve positive definiteness by construction and scale to massive neural networks.
Read aloud
Quasi-Newton QN methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes. Quasi-Newton QN methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes. We introduce SoftServe, a family of QN methods designed to overcome these obstacles without line searches or ad hoc curvature corrections. SoftServe derives positivedefinite curvature estimates from the variational objective of Berglund et al. twenty twenty five, even in the presence of negative curvature. We develop diagonal and Kroneckerfactored variants that preserve positive definiteness by construction and scale to massive neural networks.
Sources in this cluster
- SoftServe: A Scalable Quasi-Newton Method for Deep Learning
arXiv cs.AI · 2 Oct 2026, 03:58 AEST