Cosmological simulators are an important component in the study of the formation of galaxies and large scale structures, and can help answer many important questions about the universe. Despite their utility, existing parallel simulators do not scale effectively on modern machines containing thousands of processors. In this paper we present ChaNGa, a recently released production simulator based on the Charm++ infrastructure. To achieve scalable performance, ChaNGa employs various optimizations that maximize the overlap between computation and communication. We present experimental results of ChaNGa simulations on machines with thousands of processors, including the IBM Blue Gene/L and the Cray XT3. The paper goes on to highlight efforts toward even more efficient and scalable cosmological simulations. In particular, novel load balancing schemes that base decisions on certain characteristics of tree-based particle codes are discussed. Further, the multistepping capabilities of ChaNGa are presented, as are solutions to the load imbalance that such multiphase simulations face. We outline key requirements for an effective practical implementation and conclude by discussing preliminary results from simulations run with our multiphase load balancer.

Abstract-This paper focuses on the use of GPGPU-based clusters for hierarchical N -body simulations. Whereas the behavior of these hierarchical methods has been studied in the past on CPU-based architectures, we investigate key performance issues in the context of clusters of GPUs. These include kernel organization and efficiency, the balance between tree traversal and force computation work, grain size selection through the tuning of offloaded work request sizes, and the reduction of sequential bottlenecks. The effects of various application parameters are studied and experiments done to quantify gains in performance. Our studies are carried out in the context of a production-quality parallel cosmological simulator called ChaNGa. We highlight the re-engineering of the application to make it more suitable for GPU-based environments. Finally, we present performance results from experiments on the NCSA Lincoln GPU cluster, including a note on GPU use in multistepped simulations.

We explore the characteristics of actively accreting massive black holes (MBHs) within dwarf galaxies in the Romulus25 cosmological hydrodynamic simulation. We examine the MBH occupation fraction, X-ray active fractions, and active galactic nucleus (AGN) scaling relations within dwarf galaxies of stellar mass 108 M ⊙ < M star < 1010 M ⊙ out to redshift z = 2. In the local universe, the MBH occupation fraction is consistent with observed constraints, dropping below unity at M star < 3 × 1010 M ⊙, M 200 < 3 × 1011 M ⊙. Local dwarf AGN in Romulus25 follow observed scaling relations between AGN X-ray luminosity, stellar mass, and star formation rate, though they exhibit slightly higher active fractions and number densities than comparable X-ray observations. Since z = 2, the MBH occupation fraction has decreased, the population of dwarf AGN has become overall less luminous, and as a result the overall number density of dwarf AGN has diminished. We predict the existence of a large population of MBHs in the local universe with low X-ray luminosities and high contamination from X-ray binaries and the hot interstellar medium that are undetectable by current X-ray surveys. These hidden MBHs make up 76% of all MBHs in local dwarf galaxies and include many MBHs that are undermassive relative to their host galaxy’s stellar mass. Their detection relies on not only greater instrument sensitivity but also better modeling of X-ray contaminants or multiwavelength surveys. Our results indicate that dwarf AGN were substantially more active in the past, despite having low luminosity today, and that future deep X-ray surveys may uncover many hidden MBHs in dwarf galaxies out to at least z = 2.

No abstract