Managing memory and reducing I/O cost for correlation matrix calculation in bioinformatics

Abstract

Abstract-The generation of a correlation matrix from a large set of long gene sequences is a common requirement in many bioinformatics problems such as phylogenetic analysis. The generation is not only computationally intensive but also requires significant memory resources as, typically, few gene sequences can be simultaneously stored in primary memory. The standard practice in such computation is to use frequent input/output (I/O) operations. Therefore, minimizing the number of these operations will yield much faster run-times. This paper develops an approach for the faster and scalable computing of large-size correlation matrices through the full use of available memory and a reduced number of I/O operations. The approach is scalable in the sense that the same algorithms can be executed on different computing platforms with different amounts of memory and can be applied to different problems with different correlation matrix sizes. The significant performance improvement of the approach over the existing approaches is demonstrated through benchmark examples.

A distributed computing framework for All-to-All comparison problems

“…Its computation has been recently investigated for single computer platforms [17,18]. It is re-programmed for our experiments in this paper by using the Application Programming Interfaces (APIs) provided in our distributed computing framework.…”

“…All-to-All Comparisons are a key calculation stage in Multiple Sequence Alignment (MSA) and also in studying phylogenetic diversity in protein families [16]. In general, big data processing in these problems include calculation of a cross-similarity matrix between each pair of the data sequences [17,18]. This calculation is followed by several data grouping stages.…”