Stochastic Subspace Cubic Newton Method

Hanzely, Filip; Doikov, Nikita; Richtárik, Peter; Nesterov, Yurii

doi:10.48550/arxiv.2002.09526

Cited by 3 publications

(4 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…According to the estimate (18), in order to get ε-accuracy in function value, it is enough to perform…”

Section: Contracting-domain Newton Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Convex optimization based on global lower second-order models

Doikov,

Nesterov

2020

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we present new second-order algorithms for composite convex optimization, called Contracting-domain Newton methods. These algorithms are affine-invariant and based on global second-order lower approximation for the smooth component of the objective. Our approach has an interpretation both as a second-order generalization of the conditional gradient method, or as a variant of trust-region scheme. Under the assumption, that the problem domain is bounded, we prove O(1/k 2 ) global rate of convergence in functional residual, where k is the iteration counter, minimizing convex functions with Lipschitz continuous Hessian. This significantly improves the previously known bound O(1/k) for this type of algorithms. Additionally, we propose a stochastic extension of our method, and present computational results for solving empirical risk minimization problem.

show abstract

“…According to the estimate (18), in order to get ε-accuracy in function value, it is enough to perform…”

Section: Contracting-domain Newton Methodsmentioning

confidence: 99%

“…Later on, accelerated [27], adaptive [6,7] and universal [16,11,17] second-order schemes based on cubic regularization were developed. Randomized versions of Cubic Newton, suitable for solving high-dimensional problems were proposed in [12,18].…”

Section: Introductionmentioning

confidence: 99%

Convex optimization based on global lower second-order models

Doikov,

Nesterov

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Randomized subspace/projections methods have recently attracted much interest for local or convex optimization problems; see for example, [30,33,38,42,44,57]; no low effective dimensionality assumption is made in these works. Finally, we note that the main step in our convergence analysis consists in deriving a lower bound on the probability that a random subspace of given dimension intersects a given set (the set of approximate global minimizers), which is an important problem in stochastic geometry, see, e.g., the extensive discussion by Oymak and Tropp [46].…”

Section: Introductionmentioning

confidence: 99%

Bound-constrained global optimization of functions with low effective dimensionality using multiple random embeddings

2022

View full text Add to dashboard Cite

We consider the bound-constrained global optimization of functions with low effective dimensionality, that are constant along an (unknown) linear subspace and only vary over the effective (complement) subspace. We aim to implicitly explore the intrinsic low dimensionality of the constrained landscape using feasible random embeddings, in order to understand and improve the scalability of algorithms for the global optimization of these special-structure problems. A reduced subproblem formulation is investigated that solves the original problem over a random low-dimensional subspace subject to affine constraints, so as to preserve feasibility with respect to the given domain. Under reasonable assumptions, we show that the probability that the reduced problem is successful in solving the original, full-dimensional problem is positive. Furthermore, in the case when the objective’s effective subspace is aligned with the coordinate axes, we provide an asymptotic bound on this success probability that captures its polynomial dependence on the effective and, surprisingly, ambient dimensions. We then propose X-REGO, a generic algorithmic framework that uses multiple random embeddings, solving the above reduced problem repeatedly, approximately and possibly, adaptively. Using the success probability of the reduced subproblems, we prove that X-REGO converges globally, with probability one, and linearly in the number of embeddings, to an $$\epsilon $$ ϵ -neighbourhood of a constrained global minimizer. Our numerical experiments on special structure functions illustrate our theoretical findings and the improved scalability of X-REGO variants when coupled with state-of-the-art global—and even local—optimization solvers for the subproblems.

show abstract

“…Randomized subspace methods have recently attracted much interest for local or convex optimization problems; see for example, [41,36,28,31]; no low effective dimensionality assumption is made in these works. Finally, we note that the main step in our convergence analysis consists in deriving a lower bound on the probability that a random subspace of given dimension intersects a given set (the set of approximate global minimizers), which is an important problem in stochastic geometry, see, e.g., the extensive discussion by Oymak and Tropp [43].…”

Section: Introductionmentioning

confidence: 99%

Constrained global optimization of functions with low effective dimensionality using multiple random embeddings

Cartis¹,

Massart²,

Otemissov³

2020

Preprint

View full text Add to dashboard Cite

We consider the bound-constrained global optimization of functions with low effective dimensionality, that are constant along an (unknown) linear subspace and only vary over the effective (complement) subspace. We aim to implicitly explore the intrinsic low dimensionality of the constrained landscape using feasible random embeddings, in order to understand and improve the scalability of algorithms for the global optimization of these special-structure problems. A reduced subproblem formulation is investigated that solves the original problem over a random low-dimensional subspace subject to affine constraints, so as to preserve feasibility with respect to the given domain. Under reasonable assumptions, we show that the probability that the reduced problem is successful in solving the original, full-dimensional problem is positive. Furthermore, in the case when the objective's effective subspace is aligned with the coordinate axes, we provide an asymptotic bound on this success probability that captures its algebraic dependence on the effective and, surprisingly, ambient dimensions. We then propose X-REGO, a generic algorithmic framework that uses multiple random embeddings, solving the above reduced problem repeatedly, approximately and possibly, adaptively. Using the success probability of the reduced subproblems, we prove that X-REGO converges globally, with probability one, and linearly in the number of embeddings, to an -neighbourhood of a constrained global minimizer. Our numerical experiments on special structure functions illustrate our theoretical findings and the improved scalability of X-REGO variants when coupled with state-of-the-art global -and even localoptimization solvers for the subproblems.

show abstract

Stochastic Subspace Cubic Newton Method

Cited by 3 publications

References 30 publications

Convex optimization based on global lower second-order models

Convex optimization based on global lower second-order models

Bound-constrained global optimization of functions with low effective dimensionality using multiple random embeddings

Constrained global optimization of functions with low effective dimensionality using multiple random embeddings

Contact Info

Product

Resources

About