Accelerated Forward-Backward Optimization using Deep Learning

Banert, Sebastian; Rudzusika, Jevgenija; Öktem, Ozan; Adler, Jonas

doi:10.48550/arxiv.2105.05210

Cited by 5 publications

(9 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their resulting fixed points satisfy general notions of equilibrium (similar to the one arising in neural network architectures/paradigms, plug-and-play methods [21], [43], [34]) instead of being the solution to a minimization problem. In this aspect, our work is in line with the recent efforts to design [49], [48] and learn [3], [10], [43], [12] more expressive variants of well-known optimization schemes while keeping convergence guarantees.…”

Section: Introductionsupporting

confidence: 55%

Convergence Results for Primal-Dual Algorithms in the Presence of Adjoint Mismatch

Chouzenoux

Contreras

Pesquet

et al. 2023

SIAM J. Imaging Sci.

View full text Add to dashboard Cite

Most optimization problems arising in imaging science involve high-dimensional linear operators and their adjoints. In the implementations of these operators, approximations may be introduced for various practical considerations (e.g., memory limitation, computational cost, convergence speed), leading to an adjoint mismatch. This occurs for the X-ray tomographic inverse problems found in Computed Tomography (CT), where the adjoint of the measurement operator (called projector) is often replaced by a surrogate operator. The resulting adjoint mismatch can jeopardize the convergence properties of iterative schemes used for image recovery. In this paper, we study the theoretical behavior of a panel of primal-dual proximal algorithms, which rely on forward-backward-(forward) splitting schemes, when an adjoint mismatch occurs. We analyze these algorithms by focusing on the resolution of possibly non-smooth convex penalized minimization problems in an infinite-dimensional setting. By using tools from fixed point theory, we show that they can solve monotone inclusions that go beyond minimization problems. Such findings indicate these algorithms can be seen as a generalization of classical primal-dual formulations. The applicability of our findings are also demonstrated through two numerical experiments in the context of CT image reconstruction.

show abstract

Section: Introductionsupporting

confidence: 55%

Convergence Results for Primal-Dual Algorithms in the Presence of Adjoint Mismatch

Chouzenoux

Contreras

Pesquet

et al. 2023

SIAM J. Imaging Sci.

View full text Add to dashboard Cite

show abstract

“…Safeguarding is a common technique to ensure global convergence in optimization algorithms, for instance the Wolfe conditions in line-search [25,Chapter 3] ensure a sufficient decrease in the objective function value, and trust-region methods [25,Chapter 4] are based on a quadratic model having sufficient accuracy within a given radius. Recently, a norm condition similar to (4) has been combined with a deep-learning approach to speed up the convergence [4]. Even for monotone operators, line-search strategies with safeguarding have been developed, see [36,Eq.…”

Section: Introductionmentioning

confidence: 99%

“…It uses two deviation vectors and a slightly more involved safeguard condition. A similar algorithm with deviation vectors has been proposed in [4] to extend the proximal gradient method for convex minimization. The fact that we consider the more general monotone inclusion setting, allows us to apply our results, e.g., to the Chambolle-Pock [7] and Condat-Vũ [13,38] methods that both are preconditioned FB methods [20].…”

Section: Introductionmentioning

confidence: 99%

Forward-Backward Splitting with Deviations for Monotone Inclusions

Sadeghi¹,

Banert²,

Giselsson³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We propose and study a weakly convergent variant of the forwardbackward algorithm for solving structured monotone inclusion problems. Our algorithm features a per-iteration deviation vector which provides additional degrees of freedom. The only requirement on the deviation vector to guarantee convergence is that its norm is bounded by a quantity that can be computed online. This approach provides great flexibility and opens up for the design of new and improved forward-backward-based algorithms, while retaining global convergence guarantees. These guarantees include linear convergence of our method under a metric subregularity assumption without the need to adapt the algorithm parameters.Choosing suitable monotone operators allows for incorporating deviations into other algorithms, such as Chambolle-Pock and Krasnosel'skiȋ-Mann iterations. We propose a novel inertial Chambolle-Pock algorithm by selecting the deviations along a momentum direction and deciding their size using the norm condition. Numerical experiments demonstrate our convergence claims and show that even this simple choice of deviation vector can improve the performance, compared, e.g., to the standard Chambolle-Pock algorithm.

show abstract

“…• Banert et al (2020) consider theoretical foundations for data-driven nonsmooth optimization and show applications in deblurring and solving inverse problems for computed tomography and is further developed in Banert et al (2021).…”

Section: Related Workmentioning

confidence: 99%

Tutorial on amortized optimization for learning to optimize over continuous domains

Amos¹

2022

Preprint

View full text Add to dashboard Cite

Optimization is a ubiquitous modeling tool and is often deployed in settings which repeatedly solve similar instances of the same problem. Amortized optimization methods use learning to predict the solutions to problems in these settings. This leverages the shared structure between similar problem instances. In this tutorial, we will discuss the key design choices behind amortized optimization, roughly categorizing 1) models into fully-amortized and semi-amortized approaches, and 2) learning methods into regression-based and objectivebased. We then view existing applications through these foundations to draw connections between them, including for manifold optimization, variational inference, sparse coding, meta-learning, control, reinforcement learning, convex optimization, and deep equilibrium networks. This framing enables us easily see, for example, that the amortized inference in variational autoencoders is conceptually identical to value gradients in control and reinforcement learning as they both use fully-amortized models with an objective-based loss.

show abstract

Accelerated Forward-Backward Optimization using Deep Learning

Cited by 5 publications

References 25 publications

Convergence Results for Primal-Dual Algorithms in the Presence of Adjoint Mismatch

Convergence Results for Primal-Dual Algorithms in the Presence of Adjoint Mismatch

Forward-Backward Splitting with Deviations for Monotone Inclusions

Tutorial on amortized optimization for learning to optimize over continuous domains

Contact Info

Product

Resources

About