Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate
CoRR(2024)
摘要
Second-order optimization methods, such as cubic regularized Newton methods,
are known for their rapid convergence rates; nevertheless, they become
impractical in high-dimensional problems due to their substantial memory
requirements and computational costs. One promising approach is to execute
second-order updates within a lower-dimensional subspace, giving rise to
subspace second-order methods. However, the majority of existing subspace
second-order methods randomly select subspaces, consequently resulting in
slower convergence rates depending on the problem's dimension d. In this
paper, we introduce a novel subspace cubic regularized Newton method that
achieves a dimension-independent global convergence rate of
O(1/mk+1/k^2) for solving convex optimization
problems. Here, m represents the subspace dimension, which can be
significantly smaller than d. Instead of adopting a random subspace, our
primary innovation involves performing the cubic regularized Newton update
within the Krylov subspace associated with the Hessian and the gradient of the
objective function. This result marks the first instance of a
dimension-independent convergence rate for a subspace second-order method.
Furthermore, when specific spectral conditions of the Hessian are met, our
method recovers the convergence rate of a full-dimensional cubic regularized
Newton method. Numerical experiments show our method converges faster than
existing random subspace methods, especially for high-dimensional problems.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要