A Scalable Parallel Multi-Purpose Reservoir Simulator

Abstract

This paper describes the algorithms and implementation of a parallel reservoir simulator designed for, but not limited to, distributed-memory computational platforms that can solve previously prohibitive problems efficiently. The parallel simulator inherits the multipurpose features of the in-house sequential simulator, which is at the core of the new capability. As a result, black-oil, miscible, compositional, and thermal problems can be solved efficiently using this new simulator. A multilevel domain decomposition approach is used. First, the original reservoir is decomposed into several domains, each of which is given to a separate processing node. All nodes then execute computations in parallel, each node on its associated subdomain. The parallel computations include initialization, coefficient generation, linear solution on the sub-domain, and input/output. To enhance the convergence rate, we solve a coarse global problem which is generated via a multigrid-like coarsening procedure. This solution serves as a preconditioner of an outer parallel GMRES loop. The exchange of information across the subdomains, or processors, is achieved using the message passing interface standard, MPI. The use of MPI ensures portability across different computing platforms ranging from massively parallel machines to clusters of workstations. Results indicate the simulator exhibits excellent scalability of the simulator for up to 32 processors on the IBM SP2 system. Scalability results are also presented for a cluster of IBM workstations connected via an ATM (Asynchronous Transfer Mode) communication. The use of ATM for interprocessor communication was found to have a small, but measurable, impact on scaling performance. Introduction The predictive capacity of a reservoir simulator depends first on the quality of the information used, and then it depends on the ability of the computational grid and solution method to describe the flow behavior accurately. The injection of more detail into reservoir description is producing very large models. Scale up technology can be applied to reduce the overall size of the models while preserving the important details of the flow. For large-scale reservoir displacements, the scaled up model itself could consist of millions of gridblocks Flow simulation using models of that size is beyond the current capability of uniprocessor, or even shared-memory multiprocessor, compute platforms. In this work, we describe the development of a parallel multi-purpose reservoir simulator that can solve previously prohibitive problems efficiently. In addition, the parallel simulator provides the means to validate and help improve the scaled-up model by comparing its flow predictions with detailed simulations using the original finer scale description from which it is derived. In the following sections, we give a brief overview of the parallel computing landscape and we adopt a definition for scalability. That is followed by a description of our parallel simulation development strategy and implementation details. Performance results for the parallel simulator are then presented and analyzed. We close with key conclusions. Background The development of application codes for distributed-memory parallel platforms has been, until recently, a high-risk investment, both in terms of capital and manpower. This high-risk environment was due, in large part, to (1) an unstable landscape of parallel computing vendors/machines and (2) a lack of software portability across the various platforms. The focus had been on massively-parallel machines with proprietary architectures that link hundreds, or even thousands, of specially designed processors. Because the processing nodes in these machines tended to have limited computing power and small local memory, massive parallelism, both in terms of total memory and compute power, was achieved by employing thousands of such processors. P. 17^

Parallel Reservoir Simulation with OpenACC and Domain Decomposition

“…For instance, in [3], Chien et al proposed a vectorized, parallel-processed algorithm over one CRAY X-MP machine for the local grid refinement and adaptive implicit schemes in a general purpose reservoir simulator. As another example in [4], IBM SP2 machines with 32 processors have been used to achieve a scalable parallel multi-purpose reservoir simulator based on MPI techniques. Furthermore, in order to accelerate distributed reservoir simulators, modern high-performance computing platforms like GRID and Cloud were employed as well.…”

“…There has been considerable effort on parallel reservoir simulation with various high-performance techniques and platforms [2]. For example, since the 1990s multiple-core platforms, such as CRAY X-MP [3] and IBM SP2 [4], have been used to accelerate the reservoir simulation. Furthermore, modern high-performance computing platforms like GRID [5] and Cloud [6] were employed as well to address the issue of computing performance of reservoir simulations.…”

On Robust and Efficient Parallel Reservoir Simulation on Tianhe-2

“…There has been considerable effort in developing and applying parallel computing techniques in reservoir simulation since 1980s (Chien and Northrup 1993;Chien et al 1997;Killough and Bhogeswara 1991;Killough et al 1997;Li et al 1995;Parashar et al 1997;Rame and Delshad 1995;Rutledge et al 1992;Scott et al 1987;Shiralkar et al 1997;Wheeler and Smith 1990). Black oil model (Rutledge et al 1992;Wheeler and Smith 1990), compositional model (Chien et al 1997;Dogru et al 2002;Killough and Bhogeswara 1991), chemical flooding model (Rame and Delshad 1995) and dual porosity dual permeability model (Al-Shaalan et al 2003) has been numerically solved using different paradigms of parallel simulation, including shared memory, distributed memory, and GPGPUs (Dogru et al 2013) with structured grids and unstructured grids (Fung and Dogru 2008b).…”

“…Black oil model (Rutledge et al 1992;Wheeler and Smith 1990), compositional model (Chien et al 1997;Dogru et al 2002;Killough and Bhogeswara 1991), chemical flooding model (Rame and Delshad 1995) and dual porosity dual permeability model (Al-Shaalan et al 2003) has been numerically solved using different paradigms of parallel simulation, including shared memory, distributed memory, and GPGPUs (Dogru et al 2013) with structured grids and unstructured grids (Fung and Dogru 2008b). The largest reservoir model simulated has over 2 billion cells based on a compositional n-porosity formulation .…”

A numerical study of the additive Schwarz preconditioned exact Newton method (ASPEN) as a nonlinear preconditioner for immiscible and compositional porous media flow

“…State-of-the-art simulators use so-called constrained pressure residual (CPR) preconditioners [5,24,48,49], which fall into the first category and are designed to exploit the mixed elliptic-hyperbolic character of the system by first solving for a pressure-like variable by an efficient algebraic multigrid (AMG) preconditioner [10,12,45,46], followed by a broadband smoother like incomplete lowerupper factorization with zero fill-in (ILU0) applied to the whole system. Standard domain additive and multiplicative Schwarz decomposition methods have been used for decades as a parallelization strategy [6,16,18,24]. In addition, recent multiscale methods (see [27] for an overview) can be interpreted as domain decomposition methods [40]).…”