A GPU-Accelerated Linear Solver for Massively Parallel Underground Simulations

Commercial reservoir simulators have traditionally been optimized for distributed parallel execution on Central Processing Units (CPUs). Recent advances in Graphics Processing Units (GPUs) have led to the development of GPU-native simulators and triggered a shift towards a hardware-agnostic design in existing CPU solutions. For the latter, the suite of algorithms and data structures employed for a given computation are implemented for each target device. This results in a hybrid approach, where some simulator components inherently expose enough instruction parallelism or memory bandwidth requirements to warrant running on the GPU, while others are more suitable for the CPU. This paper examines the performance characteristics of a commercial black-oil reservoir simulator, which was recently extended with GPU support. Each simulation case will distribute load on the various modules in a reservoir simulator differently, depending on the target physical properties and the forecasted data desired. To assess this, the scalability of the simulator is measured in detail using the CPU and GPU, for components where both implementations are available, focusing on time spent during model initialization, property calculation, linearization, solver, field management and reporting. This is done using test cases which stress the simulator across several axes: grid resolution, different petrophysical property distributions, well count and the volume of reported data. The synthetic models which form the basis for these studies were designed to represent realistic reservoir engineering scenarios. The results show that a static partition between CPU- and GPU-assigned tasks, as employed by default in the simulator, is performant for scenarios where the work dedicated to grid cell properties and linear solution vastly outnumbers the effort spent resolving well or aquifer connections, field management and reporting. This is expected for typical simulation cases. However, when one of the latter aspects becomes dominant, the balance can shift, leading to suboptimal hardware utilization. In conclusion, if performance across all possible inputs is to be maintained, then a fully-CPU-and-GPU-capable simulator is needed, employing a dynamic scheduling strategy, where the runtime data locality, volume and parallelism of the corresponding computations are all considered when determining the target device for each operation. To the authors’ knowledge, a study on the scalability of a commercial reservoir simulator, across two different hardware architectures, has not previously been conducted to this level of detail. The results on realistic models are presented in the hope that they will contribute to the discussion surrounding the benefits of modern computing hardware for reservoir simulation and help drive deployment and design decisions for existing and future developments in both the commercial and academic spheres.

Graphics Processing Unit Performance Scalability Study on a Commercial Black-Oil Reservoir Simulator

Ţene¹,

Sekachev

Dias³

et al. 2023

A GPU-Accelerated Simulator for Challenging Extreme-Scale Geomechanical Models

Frigo¹,

Isotton²,

Janna³

et al. 2023

Simulation software is a very common tool to model geomechanical problems since direct measurements are extremely expensive and usually unfeasible. In addition, there is an increasing interest in simulating past events and forecasting future ones. Very fine meshes are needed to provide a realistic representation of complex stratigraphy. Hence the full exploitation of modern HPC infrastructure is mandatory. In this work, a fully parallel GPU-accelerated simulator for extreme-scale models is presented and its performance is assessed through a real basin scale model that is used as benchmark.

Implementing a Hardware Agnostic Commercial Black-Oil Reservoir Simulator

Szyndel¹,

Lemon²,

Dias³

et al. 2023

Commercial reservoir simulators have traditionally been optimized for parallel computations on central processing units (CPUs). The recent advances in general-purpose graphics processing units (GPUs) have provided a powerful alternative to CPU, presenting an opportunity to significantly reduce run times for simulations. Realizing peak performance on GPU requires that GPU-specific code be written, and also requires that data are laid out sympathetically to the hardware. The cost of copying data between the CPU memory and GPU memory at the time of this writing is egregious. Peak performance will only be realized if this is minimized. In paper Cao et al., 2021, the authors establish approaches to enable a simulator to give excellent performance on a CPU or GPU, with the same simulation result using either hardware. We discuss how their prototype was generalized into high-quality, maintainable code with applicability across a wide range of models. Different parts of a reservoir simulator benefit from different approaches. A modern, object-oriented simulator requires components to handle initialization, property calculation, linearization, linear solver, well and aquifer calculations, field management, and reporting. Each of these areas will present architectural challenges when broadening the scope of the simulator from CPU only to supporting CPU or GPU. We outline these challenges and present the approaches taken to address them. In particular, we discuss the importance of abstracting compute scheduling, testing methods, data storage classes, and associated memory management to a generic framework layer. We have created a high-quality reservoir simulator with the capacity to run on a CPU or GPU with results that match to within a very small tolerance. We present software engineering approaches that enable the team to achieve and maintain this in the future. In addition, we present test outcomes and discuss how to achieve excellent performance. To our knowledge, no simulator capable of both CPU simulation and full GPU simulation (meaning simulation with no copies of full grid-size data for purposes other than reporting) has been presented. We will present novel software approaches used to implement the first such commercial simulator.