HS-AUTOFIT: a highly scalable AUTOFIT application for Cloud and HPC environments

Abstract

The technological progress is leading to an increase of instrument sensitivity in the field of rotational spectroscopy. A direct consequence of such a progress is an increasing amount of data produced by instruments, for which the currently available analysis software is becoming limited and inadequate. In order to improve data analysis performance, parallel computing techniques and distributed computing technologies like Cloud and High Performance Computing (HPC) can be exploited. Despite the availability of computer resources, neither Cloud nor HPC have been fully investigated for identifying unknown target spectra in rotational spectrum. This paper proposes the design and implementation of a Highly Scalable AUTOFIT (HS-AUTOFIT), an enhanced version of a fitting tool for broadband rotational spectra that is capable of exploiting the resources offered by multiple computing nodes. Compared to the old program version, the new one is capable of scaling on multiple computing nodes, thus guaranteeing higher accuracy of the fit function and consistent boost of execution time. The result of tests conducted in real Cloud and HPC environments show that HS-AUTOFIT is a viable solution for the analysis of huge amount of data in the addressed scientific field.

Testing the Scalability of the HS-AUTOFIT Tool in a High-Performance Computing Environment

“…In [6], we proposed HS-AUTOFIT, an enhanced and parallel version of the original AUTOFIT algorithm that exploits the computing power of multiple resources to speed up the completion time of highly demanding jobs. Specifically, we implemented two versions of HS-AUTOFIT that can execute on an HPC cluster and on Cloud-provided virtual machines, respectively.…”

“…We propose to overcome such a limitation by leveraging the scale power typically offered by dense computing contexts exploiting HPC technology. To this end, in [6] we proposed HS-AUTOFIT, a modified version of AUTOFIT capable of scaling out to multiple CPU cores of an HPC cluster of nodes. Specifically, to support the distribution of tasks among the cores of an HPC cluster, HS-AUTOFIT offers the following services: transfer of generic data types and small-sized data (e.g., configuration parameters); transfer/sharing of text files (containing the triplets to fit); and support for a synchronized communication mode.…”

“…In the tests, we considered the following workloads: 7957 triplets, 39,556 triplets, and 231,880 triplets. For each workload, we ran HS-AUTOFIT instances with increasing numbers of cores in the range [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16]. In Figure 5a,b, we report the results of the tests in the form of box-and-whisker charts for the workloads of 7957 triplets and 231,880 triplets, respectively.…”

“…To overcome this limitation, in a previous work [6] we proposed a new version of AUTOFIT (HS-AUTOFIT) capable of scaling out on multiple machines. This enhanced version of the application leverages the computing power of cluster nodes to offer theoretically unbounded horizontal scalability.…”