Monte Carlo domain decomposition for robust nuclear reactor analysis

Abstract

Monte Carlo (MC) neutral particle transport codes are considered the gold-standard for nuclear simulations, but they cannot be robustly applied to high-fidelity nuclear reactor analysis without accommodating several terabytes of materials and tally data. While this is not a large amount of aggregate data for a typical high performance computer, MC methods are only embarrassingly parallel when the key data structures are replicated for each processing element, an approach which is likely infeasible on future machines. The present work explores the use of spatial domain decomposition to make full-scale nuclear reactor simulations tractable with Monte Carlo methods, presenting a simple implementation in a production-scale code. Good performance is achieved for mesh-tallies of up to 2.39TB distributed across 512 compute nodes while running a full-core reactor benchmark on the Mira Blue Gene/Q supercomputer at the Argonne National Laboratory. In addition, the effects of load imbalances are explored with an updated performance model that is empirically validated against observed timing results. Several load balancing techniques are also implemented to demonstrate that imbalances can be largely mitigated, including a new and efficient way to distribute extra compute resources across coarse domain meshes.

Enhancing Monte Carlo Particle Transport for Modern Many-Core Architectures

“…Similarly, Horelik, Siegel, Forget, and Smith (2014) explored several spatial domain decomposition methods and analyzed their effect on load imbalance. In summary, each of these groups identified load imbalance as a problem and proposed analysis and solutions that fit their specific needs.…”

ARRC: A random ray neutron transport code for nuclear reactor simulation

“…Any method aiming to simulate entire nuclear reactors in high fidelity must be able to map efficiently onto supercomputer architectures, where many separate computers are networked together and working in tight coordination to run a simulation [34,35,36]. Running on a supercomputer is necessary both to reduce the time to solution and to greatly increase the total amount of memory available for the simulation in order to allow a 3D full reactor core to be simulated with a sufficiently high resolution [18,37,38,39,40,41] . In order for a MOC or discrete ordinance (Sn) based simulation to utilize a supercomputer effectively, the domain of the reactor must be decomposed and distributed among the hundreds or thousands of computational nodes that form the supercomputer [25,26,29,30,31].…”

“…For instance, the transfer of rays between adjacent computational domains inside of a power iteration in TRRM is similar to the transfer of particles between domains in a domain decomposed MC simulation. In recent years there has been significant research into these MC communication models and costs [38,41,45,46,47,48] that were leveraged when developing the communication model used in ARRC. However, there are some key differences in communication requirements between Monte Carlo and TRRM that can sometimes lead to domain replication being preferable in Monte Carlo instead of domain decomposition (particularly when the geometry model and all tally data are small enough to fit on a single computational node).…”

Windowed multipole for cross section Doppler broadening

“…Despite such advancements, high fidelity nuclear reactor simulations are still beyond immediate reach when one considers the need to account for temperature variations during operation and the necessity of tallying reaction rates over millions of regions for tracking core evolution. As part of the CASL project, progress has been made on the last front by using data decomposition [2], overlapping domain decomposition [3] or combinations of non-overlapping domain decomposition and replication [4].…”