Investigating Bi-Level Optimization for Learning and Vision From a Unified Perspective: A Survey and Beyond

Liu, Risheng; Gao, Jiaxin; Zhang, Jin; Meng, Deyu; Lin, Zhouchen

doi:10.1109/tpami.2021.3132674

Cited by 81 publications

(36 citation statements)

References 106 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hyper-parameter optimization (Lorraine and Duvenaud 2018;Okuno, Takeda, and Kawana 2018;Franceschi et al 2018) uses bilevel optimization extensively. Besides, the idea of bilevel optimization has also been applied to meta learning (Zintgraf et al 2019;Song et al 2019;Soh, Cho, and Cho 2020), neural architecture search (Liu, Simonyan, and Yang 2018;Wong et al 2018;Xu et al 2019), adversarial learning (Tian et al 2020;Gao et al 2020), deep reinforcement learning (Tschiatschek et al 2019), sparse learning Huang 2019, 2020;Poon and Peyré 2021) etc. For a more thorough review of these applications, please refer to the Table 2 of the survey paper by Liu et al (2021).…”

Section: Related Workmentioning

confidence: 99%

A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse

Huang

2022

AAAI

View full text Add to dashboard Cite

In this paper, we propose a novel Hessian inverse free Fully Single Loop Algorithm (FSLA) for bilevel optimization problems. Classic algorithms for bilevel optimization admit a double loop structure which is computationally expensive. Recently, several single loop algorithms have been proposed with optimizing the inner and outer variable alternatively. However, these algorithms not yet achieve fully single loop. As they overlook the loop needed to evaluate the hyper-gradient for a given inner and outer state. In order to develop a fully single loop algorithm, we first study the structure of the hyper-gradient and identify a general approximation formulation of hyper-gradient computation that encompasses several previous common approaches, e.g. back-propagation through time, conjugate gradient, etc. Based on this formulation, we introduce a new state variable to maintain the historical hyper-gradient information. Combining our new formulation with the alternative update of the inner and outer variables, we propose an efficient fully single loop algorithm. We theoretically show that the error generated by the new state can be bounded and our algorithm converges. Finally, we verify the efficacy our algorithm empirically through multiple bilevel optimization based machine learning tasks. A long version of this paper can be found in: https://arxiv.org/abs/2112.04660.

show abstract

Section: Related Workmentioning

confidence: 99%

A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse

Huang

2022

AAAI

View full text Add to dashboard Cite

show abstract

“…Here the UL and LL objectives are smooth functions with Lipschitz continuous first and second order derivatives. 1…”

Section: Bilevel Alternating Gradient With Dual Correctionmentioning

confidence: 99%

“…Now we can uniformly understand the existing explicit and implicit Gradient-based BLOs. From the viewpoint of non-asymptotic convergence analysis, roughly speaking, the existing BLO methods can be divided into three steps: (1) given UL variable, update the LL variable by solving the LL optimization problem up to tolerance, which satisfies prescribed dynamic accuracy requirements. From the view point of the KKT condition, this step is to guarantee the feasible condition; (2) given UL and LL variable, update the dual multipliers by approximating the correct dual multipliers accurately.…”

Section: A Single-level Reformulation and Dual Correctionmentioning

confidence: 99%

“…Bi-Level Optimization (BLO) has recently attracted growing interests due to its wide applications in the field of deep learning [1], especially for hyperparameter optimization [2,3,4], meta learning [5,6,7], neural architecture search [8,9,10], adversarial learning [11], and reinforcement learning [12], etc. BLO tackles nested optimization structures appearing in applications.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards Extremely Fast Bilevel Optimization with Self-governed Convergence Guarantees

Liu¹,

Li²,

Yao³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Gradient methods have become mainstream techniques for Bi-Level Optimization (BLO) in learning and vision fields. The validity of existing works heavily relies on solving a series of approximation subproblems with extraordinarily high accuracy. Unfortunately, to achieve the approximation accuracy requires executing a large quantity of time-consuming iterations and computational burden is naturally caused. This paper is thus devoted to address this critical computational issue. In particular, we propose a single-level formulation to uniformly understand existing explicit and implicit Gradient-based BLOs (GBLOs). This together with our designed counter-example can clearly illustrate the fundamental numerical and theoretical issues of GBLOs and their naive accelerations. By introducing the dual multipliers as a new variable, we then establish Bilevel Alternating Gradient with Dual Correction (BAGDC), a general framework, which significantly accelerates different categories of existing methods by taking specific settings. A striking feature of our convergence result is that, compared to those original unaccelerated GBLO versions, the fast BAGDC admits a unified non-asymptotic convergence theory towards stationarity. A variety of numerical experiments have also been conducted to demonstrate the superiority of the proposed algorithmic framework.Preprint. Under review.

show abstract

“…There is also some research specifically focused on the bi-level optimization problem. See the survey of bi-level optimization algorithms [102] for a full description. The development of bi-level optimization can facilitate the research of robust SSL.…”

Section: Label Distribution Mismatchmentioning

confidence: 99%

Robust Deep Semi-Supervised Learning: A Brief Introduction

Guo¹,

Zhou²,

Li³

2022

Preprint

View full text Add to dashboard Cite

Semi-supervised learning (SSL) is the branch of machine learning that aims to improve learning performance by leveraging unlabeled data when labels are insufficient. Recently, SSL with deep models has proven to be successful on standard benchmark tasks. However, they are still vulnerable to various robustness threats in real-world applications as these benchmarks provide perfect unlabeled data, while in realistic scenarios, unlabeled data could be corrupted. Many researchers have pointed out that after exploiting corrupted unlabeled data, SSL suffers severe performance degradation problems. Thus, there is an urgent need to develop SSL algorithms that could work robustly with corrupted unlabeled data. To fully understand robust SSL, we conduct a survey study. We first clarify a formal definition of robust SSL from the perspective of machine learning. Then, we classify the robustness threats into three categories: i) distribution corruption, i.e., unlabeled data distribution is mismatched with labeled data; ii) feature corruption, i.e., the features of unlabeled examples are adversarially attacked; and iii) label corruption, i.e., the label distribution of unlabeled data is imbalanced. Under this unified taxonomy, we provide a thorough review and discussion of recent works that focus on these issues. Finally, we propose possible promising directions within robust SSL to provide insights for future research.

show abstract

Investigating Bi-Level Optimization for Learning and Vision From a Unified Perspective: A Survey and Beyond

Cited by 81 publications

References 106 publications

A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse

A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse

Towards Extremely Fast Bilevel Optimization with Self-governed Convergence Guarantees

Robust Deep Semi-Supervised Learning: A Brief Introduction

Contact Info

Product

Resources

About