Optimal checkpointing for adjoint multistage time-stepping schemes

Zhang, Hong; Constantinescu, Emil M.

doi:10.1016/j.jocs.2022.101913

Cited by 8 publications

(3 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Griewank and Walther (2000) and Symes (2007) presented optimal checkpointing, which avoids excessive memory usage by balancing I/O and computational overhead optimally. This approach, which was initially developed for generic adjoint-state methods on CPUs (Griewank & Walther, 2000), has been used successfully in seismic imaging (Symes, 2007) and machine learning (Chen et al, 2016) and has recently been extended to multi-stage timestepping (Zhang & Constantinescu, 2022). By adding on-the-fly compression and decompression of the checkpointed forward wavefields, Kukreja et al (2020) further reduced the computational overhead of optimal checkpointing.…”

Section: Introductionmentioning

confidence: 99%

Wave‐based inversion at scale on graphical processing units with randomized trace estimation

Louboutin

Herrmann

2023

Geophysical Prospecting

View full text Add to dashboard Cite

Thanks to continued performance improvements in software and hardware, wave‐equation‐based imaging technologies, such as full‐waveform inversion and reverse‐time migration, are becoming more commonplace. However, widespread adaptation of these advanced imaging modalities has not yet materialized because current implementations are not able to reap the full benefits from accelerators, in particular those offered by memory‐scarce graphics processing units. Through the use of randomized trace estimation, we overcome the memory bottleneck of this type of hardware. At the cost of limited computational overhead and controllable incoherent errors in the gradient, the memory footprint of adjoint‐state methods is reduced drastically. Thanks to this relatively simple to implement memory reduction via an approximate imaging condition, we are able to benefit from graphics processing units without memory offloading. We demonstrate the performance of the proposed algorithm on acoustic two‐ and three‐dimensional full‐waveform inversion examples and on the formation of image gathers in transverse tilted isotropic media.

show abstract

Section: Introductionmentioning

confidence: 99%

Wave‐based inversion at scale on graphical processing units with randomized trace estimation

Louboutin

Herrmann

2023

Geophysical Prospecting

View full text Add to dashboard Cite

show abstract

“…This allows traditional DRAM to be used as an extra cache level (Wu et al 2017, Stanzione et al 2020. The last mentioned approach, trading computation for storage, includes checkpointing methods (Griewank and Walther 1997, Wang et al 2009, Zhang and Constantinescu 2023 and sparse decomposition (Zhao et al 2020). More generally, this means storing the inputs required to regenerate the desired output rather than storing the output directly (Akturk and Karpuzcu 2018).…”

Section: Introductionmentioning

confidence: 99%

An autoencoder compression approach for accelerating large-scale inverse problems

Wittmer,

Badger,

Sundar

et al. 2023

Inverse Problems

View full text Add to dashboard Cite

PDE-constrained inverse problems are some of the most challenging and computationally demanding problems in computational science today. Fine meshes that are required to accurately compute the PDE solution introduce an enormous number of parameters and require large scale computing resources such as more processors and more memory to solve such systems in a reasonable time. For inverse problems constrained by time dependent PDEs, the adjoint method that is often employed to efﬁciently compute gradients and higher order derivatives requires solving a time-reversed, so-called adjoint PDE that depends on the forward PDE solution at each timestep. This necessitates the storage of a high dimensional forward solution vector at every timestep. Such a procedure quickly exhausts the available memory resources. Several approaches that trade additional computation for reduced memory footprint have been proposed to mitigate the memory bottleneck, including checkpointing and compression strategies. In this work, we propose a close-to-ideal scalable compression approach using autoencoders to eliminate the need for checkpointing and substantial memory storage, thereby reducing both the time-to-solution and memory requirements. We compare our approach with checkpointing and an off-the-shelf compression approach on an earth-scale ill-posed seismic inverse problem. The results verify the expected close-to-ideal speedup for both the gradient and Hessianvector product using the proposed autoencoder compression approach. To highlight the usefulness of the proposed approach, we combine the autoencoder compression with the data-informed active subspace (DIAS) prior to show how the DIAS method can be affordably extended to large scale problems without the need of checkpointing and large memory.

show abstract

“…Griewank & Walther (2000) proposed a checkpointing algorithm, which is optimal under certain assumptions, including that the number of steps is known in advance, and that all the storage has equal access cost. Subsequent authors have produced checkpointing algorithms that relax these requirements in various ways, such as by accounting for different types of storage (e.g., memory and disk) or by not requiring the number of steps to be known in advance, for example Stumm & Walther (2009), Aupy et al (2016), Schanen et al (2016), Aupy & Herrmann (2017), Herrmann & Pallez (2020), James R. Maddison (2023), and Zhang & Constantinescu (2023).…”

mentioning

confidence: 99%

checkpoint_schedules: schedules for incremental checkpointing of adjoint simulations

Dolci,

Maddison,

Ham

et al. 2024

JOSS

View full text Add to dashboard Cite

checkpoint_schedules provides schedules for step-based incremental checkpointing of the adjoints to computer models. The schedules contain instructions indicating the sequence of forward and adjoint steps to be executed, and the data storage and retrieval to be performed. These instructions are independent of the model implementation, which enables the model authors to switch between checkpointing algorithms without recoding. Conversely, checkpoint-ing_schedules provides developers of checkpointing algorithms a direct mechanism to convey their work to model authors. checkpointing_schedules has been integrated into tlm_adjoint (James R. Maddison et al., 2019), a Python library designed for the automated derivation of higher-order tangent-linear and adjoint models and work is ongoing to integrate it with pyadjoint (Mitusch et al., 2019). This package can be incorporated into other gradient solvers based on adjoint methods, regardless of the specific approach taken to generate the adjoint model.

show abstract

Optimal checkpointing for adjoint multistage time-stepping schemes

Cited by 8 publications

References 15 publications

Wave‐based inversion at scale on graphical processing units with randomized trace estimation

Wave‐based inversion at scale on graphical processing units with randomized trace estimation

An autoencoder compression approach for accelerating large-scale inverse problems

checkpoint_schedules: schedules for incremental checkpointing of adjoint simulations

Contact Info

Product

Resources

About