One-Shot Approaches to Design Optimzation

Bosse, Torsten; Gauger, Nicolas R.; Griewank, Andreas; Günther, Stefanie; Schulz, Volker

doi:10.1007/978-3-319-05083-6_5

Cited by 25 publications

(19 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The MGRIT algorithm replaces the O(N ) sequential time stepping algorithm with a highly parallel O(N ) multigrid algorithm [56]. Essentially, a sequence of coarser temporal grids are used to accelerate the solution of the fine grid (5). The MGRIT algorithm easily extends to nonlinear problems with the Full Approximation Storage (FAS) nonlinear multigrid method [6], which is the same nonlinear multigrid cycling used by PFASST [10].…”

Section: Multigrid Reduction In Time Using Xbraidmentioning

confidence: 99%

A non-intrusive parallel-in-time adjoint solver with the XBraid library

Günther

Gauger

Schroder

2018

Comput. Visual Sci.

Self Cite

View full text Add to dashboard Cite

In this paper, an adjoint solver for the multigrid-in-time software library XBraid is presented. XBraid provides a non-intrusive approach for simulating unsteady dynamics on multiple processors while parallelizing not only in space but also in the time domain [60]. It applies an iterative multigrid reduction in time algorithm to existing spatially parallel classical time propagators and computes the unsteady solution parallel in time. Techniques from Automatic Differentiation are used to develop a consistent discrete adjoint solver which provides sensitivity information of output quantities with respect to design parameter changes. The adjoint code runs backwards through the primal XBraid actions and accumulates gradient information parallel in time. It is highly non-intrusive as existing adjoint time propagators can easily be integrated through the adjoint interface. The adjoint code is validated on advection-dominated flow with periodic upstream boundary condition. It provides similar strong scaling results as the primal XBraid solver and offers great potential for speeding up the overall computational costs for sensitivity analysis using multiple processors.

show abstract

Section: Multigrid Reduction In Time Using Xbraidmentioning

confidence: 99%

A non-intrusive parallel-in-time adjoint solver with the XBraid library

Günther

Gauger

Schroder

2018

Comput. Visual Sci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the preceding section, we showed that there exist stepsizes α G and (projected Newton) preconditioners C that guarantee that the extended mapping G satisfies the contraction condition (2) and, thus, allow convergence of the overall method. A necessary condition for their existence was that there is only a slight coupling of the variables by the constraints, namely, cG or dG are sufficiently small.…”

Section: Enforcing Contraction For the General Casementioning

confidence: 97%

Augmenting the one-shot framework by additional constraints

Bosse

2016

Optimization Methods and Software

Self Cite

View full text Add to dashboard Cite

The (multistep) one-shot method for design optimization problems has been successfully implemented for various applications. To this end, a slowly convergent primal fixed-point iteration of the state equation is augmented by an adjoint iteration and a corresponding preconditioned design update. In this paper we present a modification of the method that allows for additional equality constraints besides the usual state equation. A retardation analysis and the local convergence of the method in terms of necessary and sufficient conditions are given, which depend on key characteristics of the underlying problem and the quality of the utilized preconditioner.

show abstract

“…• Hessian approximation: In order to prove convergence of the simultaneous One-shot method on a theoretical level, the preconditioners B θ , B W , B µ should approximate the Hessian of an augmented Lagrangian function that involves the residual of the state and adjoint equations (see [9] and references therein). Numerically, we approximate the Hessian through consecutive limited-memory BFGS updates based on the current reduced gradient (thus assuming that the residual term is small).…”

Section: Algorithm 2 Simultaneous Layer-parallel Trainingmentioning

confidence: 99%

“…They aim at solving the optimization problem in an all-at-once fashion, updating the optimization parameters simultaneously while solving for the time-dependent system state. Here, we apply the One-shot method [9,32] to solve the training problem simultaneously for the network state and parameters. In this approach, network parameter updates are based on inexact gradient information resulting from early stopping of the layer-parallel multigrid iteration.…”

Section: Introductionmentioning

confidence: 99%

Layer-Parallel Training of Deep Residual Neural Networks

Günther¹,

Ruthotto²,

Schroder³

et al. 2020

SIAM Journal on Mathematics of Data Science

Self Cite

View full text Add to dashboard Cite

Residual neural networks (ResNets) are a promising class of deep neural networks that have shown excellent performance for a number of learning tasks, e.g., image classification and recognition. Mathematically, ResNet architectures can be interpreted as forward Euler discretizations of a nonlinear initial value problem whose time-dependent control variables represent the weights of the neural network. Hence, training a ResNet can be cast as an optimal control problem of the associated dynamical system. For similar time-dependent optimal control problems arising in engineering applications, parallel-in-time methods have shown notable improvements in scalability. This paper demonstrates the use of those techniques for efficient and effective training of ResNets. The proposed algorithms replace the classical (sequential) forward and backward propagation through the network layers by a parallel nonlinear multigrid iteration applied to the layer domain. This adds a new dimension of parallelism across layers that is attractive when training very deep networks. From this basic idea, we derive multiple layer-parallel methods. The most efficient version employs a simultaneous optimization approach where updates to the network parameters are based on inexact gradient information in order to speed up the training process. Using numerical examples from supervised classification, we demonstrate that the new approach achieves similar training performance to traditional methods, but enables layerparallelism and thus provides speedup over layer-serial methods through greater concurrency. in particular deep residual networks (ResNets) [36], have been breaking human records in various contests and are now central to technology such as image recognition [38,43,45] and natural language processing [6,15,41].The abstract goal of machine learning is to model a function f :for input-output pairs (y, c) from a certain data set Y × C. Depending on the nature of inputs and outputs, the task can be regression or classification. When outputs are available for all samples, parts of the samples, or are not available, this formulation describes supervised, semi-supervised, and unsupervised learning, respectively. The function f can be thought of as an interpolation or approximation function.In deep learning, the function f involves a DNN that aims at transforming the input data using many layers. The layers successively apply affine transformations and element-wise nonlinearities that are parametrized by the network parameters θ. The training problem consists of finding the parameters θ such that (1.1) is satisfied for data elements from a training data set, but also holds for previously unseen data from a validation data set, which has not been used during training. The former objective is commonly modeled as an expected loss and optimization techniques are used to find the parameters that minimize the loss.Despite rapid methodological developments, compute times for training state-of-the-art DNNs can still be prohibitive, measured in the orde...

show abstract

One-Shot Approaches to Design Optimzation

Cited by 25 publications

References 42 publications

A non-intrusive parallel-in-time adjoint solver with the XBraid library

A non-intrusive parallel-in-time adjoint solver with the XBraid library

Augmenting the one-shot framework by additional constraints

Layer-Parallel Training of Deep Residual Neural Networks

Contact Info

Product

Resources

About