We propose a bilevel optimization strategy for selecting the best hyperparameter value for the nonsmooth ℓ p regularizer with 0 < p ≤ 1. The concerned bilevel optimization problem has a nonsmooth, possibly nonconvex, ℓ p -regularized problem as the lower-level problem. Despite the recent popularity of nonconvex ℓ p regularizer and the usefulness of bilevel optimization for selecting hyperparameters, algorithms for such bilevel problems have not been studied because of the difficulty of ℓ p regularizer. We first show new optimality conditions for such bilevel optimization problems and then propose a smoothing-type algorithm together with convergence analysis. The proposed algorithm is simple and scalable as our numerical comparison to Bayesian optimization and grid search indicates. It is a promising algorithm for nonsmooth nonconvex bilevel optimization problems as the first algorithm with convergence guarantee.

No abstract

In this paper, we consider a nonlinear semi-infinite program that minimizes a function including a log-determinant (logdet) function over positive definite matrix constraints and infinitely many convex inequality constraints, called SIPLOG for short. The main purpose of the paper is to develop an algorithm for computing a Karush-Kuhn-Tucker (KKT) point for the SIPLOG efficiently. More specifically, we propose an interior point sequential quadratic programming-type method that inexactly solves a sequence of semi-infinite quadratic programs approximating the SIPLOG. Furthermore, to generate a search direction in the dual matrix space associated with the semi-definite constraint, we solve scaled Newton equations that yield the family of Monteiro-Zhang directions. We prove that the proposed method weakly* converges to a KKT point under some mild assumptions. Finally, we conduct some numerical experiments to demonstrate the efficiency of the proposed method.Keyword: semi-infinite program, log-determinant, nonlinear semi-definite program, sequential quadratic programming method, exchange method

No abstract