The parallelism motifs of genomic data analysis

Abstract

<jats:p>Genomic datasets are growing dramatically as the cost of sequencing continues to decline and small sequencing devices become available. Enormous community databases store and share these data with the research community, but some of these genomic data analysis problems require large-scale computational platforms to meet both the memory and computational requirements. These applications differ from scientific simulations that dominate the workload on high-end parallel systems today and place different requirements on programming support, software libraries and parallel architectural design. For example, they involve irregular communication patterns such as asynchronous updates to shared data structures. We consider several problems in high-performance genomics analysis, including alignment, profiling, clustering and assembly for both single genomes and metagenomes. We identify some of the common computational patterns or ‘motifs’ that help inform parallelization strategies and compare our motifs to some of the established lists, arguing that at least two key patterns, sorting and hashing, are missing.</jats:p> <jats:p>This article is part of a discussion meeting issue ‘Numerical algorithms for high-performance computational science’.</jats:p>

Distributed-Memory k-mer Counting on GPUs

“…Our GPU optimizations effectively turn a compute-bound problem into one dominated by communication. In particular, many-to-many k-mer exchange for redistributing k-mers tends to be the secondary bottleneck at small scales and the primary bottleneck at large scale of distributed memory kmer counters [7], [10], [22], [33]. Our novel use of supermers in distributed memory parallelization is combined with GPU optimizations, improving communication costs by reducing communication volume.…”

“…In particular, the distribution of k-mers is not fixed across biological input datasets and cannot be determined until the run time. The primary methods for scalable-distributed memory k-mer counting rely on distributed hash tables [6], [7], [10], [12], [21], [33].…”

Communication-Avoiding and Memory-Constrained Sparse Matrix-Matrix Multiplication at Extreme Scale

“…Our algorithm will boost many applications in genomics, scientific computing, and social network analysis where SpGEMM has emerged as a key computational kernel. For example, Yelick et al [39] regarded SpGEMM as a parallelism motif of genomic data analysis with applications in alignment, profiling, clustering and assembly for both single genomes and metagenomes. With the size of genomic data growing exponentially, extreme-scale SpGEMM presented in this paper will enable rapid scientific discoveries in these applications.…”

Toward a Predictive Understanding of Cyanobacterial Harmful Algal Blooms through AI Integration of Physical, Chemical, and Biological Data

“…Memory needs can approach the terabyte scale, and therefore, the computations commonly use compute clusters with >32 central processing units (CPUs) and >4 GB of random access memory per CPU. GPU computing and edge computing are popularly used due to the inherent single instruction multiple data (SIMD) nature of the computation . Download and storage of the data can also require petabyte-scale systems such as those hosted by the National Microbiome Data Collaborative …”