Optimizing fine-grained communication in a biomolecular simulation application on Cray XK6

Abstract

Abstract-Achieving good scaling for fine-grained communication intensive applications on modern supercomputers remains challenging. In our previous work, we have shown that such an application -NAMD -scales well on the full Jaguar XT5 without long-range interactions; Yet, with them, the speedup falters beyond 64K cores. Although the new Gemini interconnect on Cray XK6 has improved network performance, the challenges remain, and are likely to remain for other such networks as well. We analyze communication bottlenecks in NAMD and its CHARM++ runtime, using the Projections performance analysis tool. Based on the analysis, we optimize the runtime, built on the uGNI library for Gemini. We present several techniques to improve the fine-grained communication. Consequently, the performance of running 92224-atom Apoa1 with GPUs on TitanDev is improved by 36%. For 100-million-atom STMV, we improve upon the prior Jaguar XT5 result of 26 ms/step to 13 ms/step using 298,992 cores on Jaguar XK6.

Mapping to Irregular Torus Topologies and Other Techniques for Petascale Biomolecular Simulation

“…Stampede compares well to the Cray platforms considering that the ibverbs Charm++ network layer is seldom exercised at these node counts while the GNI network layer has been extensively tuned for petascale simulations [10], [11]. Stampede has 6,400 nodes, which are connected by an FDR InfiniBand fat tree network of eight core-switches and over 320 leaf switches with a 5/4 oversubscription, but the largest job size supported in normal operation is 2048 nodes.…”

“…PAMI [7], [8], uGNI [9], [10], [11], and IB-Verbs [12] are used on IBM Blue Gene/Q, Cray XE6/XK7/XC30, and InfiniBand clusters respectively for native builds. Charm++ can also be built on top of MPI on any system.…”

“…Mei et al showed the utility of the node-awareness techniques deployed in NAMD to scale it to the full Jaguar PF Cray XT5 (224, 076 cores) [17]. Building on that work, Sun et al explored several techniques for improving fine-grained communication on uGNI [11]. For the 100-million-atom STMV system, they improved the time per step from 26 ms/step on Jaguar XT5 to 13 ms/step using 298, 992 cores of Jaguar XK6.…”

Optimisation of a Molecular Dynamics Simulation of Chromosome Condensation

“…Perhaps the most well known molecular dynamics program applicable to biological research is NAMD [2], which has been heavily optimised for a variety of highly parallel systems over the last 20 years [3], [4], [5]. Other well known molecular dynamics codes include LAMMPS [6], DL POLY [7] and GROMACS [8].…”

CommGram: A New Visual Analytics Tool for Large Communication Trace Data

“…Sun et al [8] used the Projections performance analysis tool to examine and optimize fine-grained communication in a biomolecular simulation application. They achieved a considerable improvement on a Cray XK6 system.…”