Modelling fracture in heterogeneous materials on HPC systems using a hybrid MPI/Fortran coarray multi-scale CAFE framework

Abstract

A 3D multi-scale cellular automata finite element (CAFE) framework for modelling fracture in heterogeneous materials is described. The framework is implemented in a hybrid MPI/Fortran coarray code for efficient parallel execution on HPC platforms. Two open source BSD licensed libraries developed by the authors in modern Fortran were used: CGPACK, implementing cellular automata (CA) using Fortran coarrays, and ParaFEM, implementing finite elements (FE) using MPI. The framework implements a two-way concurrent hierarchical information exchange between the structural level (FE) and the microstructure (CA). MPI to coarrays interface and data structures are described. The CAFE framework is used to predict transgranular cleavage propagation in a polycrystalline iron round bar under tension. Novel results enabled by this CAFE framework include simulation of progressive cleavage propagation through individual grains and across grain boundaries, and emergence of a macro-crack from merging of cracks on preferentially oriented cleavage planes in individual crystals. Nearly ideal strong scaling up to at least tens of thousands of cores was demonstrated by CGPACK and by ParaFEM in isolation in prior work on Cray XE6. Cray XC30 and XC40 platforms and CrayPAT profiling were used in this work. Initially the strong scaling limit of hybrid CGPACK/ParaFEM CAFE model was 2,000 cores. After replacing all-to-all communication patterns with the nearest neighbour algorithms the strong scaling limit on Cray XC30 was increased to 7,000 cores. TAU profiling on non-Cray systems identified deficiencies in Intel Fortran 16 optimisation of remote coarray operations. Finally, coarray synchronisation challenges and opportunities for thread parallelisation in CA are discussed.

A massively parallel multiscale CAFE framework for the modelling of fracture in heterogeneous materials under dynamic loading

“…A key advantage of the framework is its highly parallel implementation. In 2018 Shterenlikht et al [34] highlighted the applicability It is therefore suitable to highlight the numerical method- [35]. Each mini-app [36] 131 solves a specific engineering problem, some examples in-132 clude: heat transfer [37], stochastic finite element mod-133 elling [38], geometric nonlinearity [39], material nonlin-134 earity [40], image-based modelling [41] and fluid structure 135 interaction [42].…”

A generic computational model for three-dimensional fracture and fragmentation problems of quasi-brittle materials

“…In the future, it is worth exploring different numerical methods to ease the requirements on time-steps. In order to boost the simulation speed, one possible solution is parallel computing (Owen and Feng, 2001;Frantík et al, 2013;Shterenlikht et al, 2018). Another possible solution is the multiscale method, which can reduce computational cost for large-scale problems (Beex et al, 2014).…”

Toward Modern Fortran Tooling and a Thriving Developer Community

“…Reid, Long, and Steidel (2020) documented the evolution of Fortran's parallel features. Although these features have seen relatively slow implementation by the compiler vendors, applications that use native parallel features are becoming more common (e.g., Garain, Balsara, and Reid 2015;Mozdzynski, Hamrud, and Wedi 2015;Shterenlikht, Margetts, and Cebamanos 2018;Curcic 2019;Diaz et al 2021). Today, Fortran remains the dominant language used on top High Performance Computing (HPC) systems, with applications to weather and climate (Powers et al 2017;Skamarock, Ong, and Klemp 2021), computational chemistry ( Čertík, Pask, and Vackář 2013;Apra et al 2020), computational fluid dynamics (Sharma and Moulitsas 2017), life science (Aguilar et al 2018;Vandenplas et al 2020), and economics (Harrison and Pearson 1996;Fehr and Kindermann 2018).…”