Parallelizing the execution of native data mining algorithms for computational biology

Abstract

SUMMARYData mining is being increasingly used in biology. Biologists are adopting prototyping languages, like R and Matlab, to facilitate the application of data mining algorithms to their data. As a result, their scripts are becoming increasingly complex and also require frequent updates. Application to large datasets becomes impractical and the time-to-paper increases. Furthermore, even if there are various systems that can be used to efficiently process large datasets, for example, using Cloud and High Performance Computing, they usually require procedures to be translated into specific languages or to be adapted to a certain computing platform. Such modifications can speed up the processing, but translation is not automatic, especially in complex cases, and can require a large amount of programming effort and accurate validation. In this paper, we propose an approach to parallelize data mining procedures in the form of compiled software or R scripts developed by biology communities of practice. Our approach requires minimal alteration of the original code. In many cases, there is no need for code modification. Furthermore, it allows for fast updating when a new version is ready. We clarify the constraints and the benefits of our method and report a practical use case to demonstrate such benefits compared with a standard execution. Our approach relies on a distributed network of web services and ultimately exposes the algorithms as-a-Service, to be invoked by remote thin clients.

Retrieving taxa names from large biodiversity data collections using a flexible matching workflow

“…On the other hand, also other taxa matching systems could benefit from the same facilities by following integration guidelines (Coro and Italiano, 2012). We give further details about the advantages and the modalities of integrating parallelisable algorithms in iMarine in the paper by Coro et al (2014). Another advantage of using the BiOnym instance on iMarine, is that access policies to authority files are managed by the eInfrastructure agreements with the data providers.…”

Cloud computing in a distributed e‐infrastructure using the web processing service standard

“…For example, several systems stress on low maintenance, low deployment costs, and high sustainability, granting reliability, easiness of installation, and usability at the same time. On the other hand, aspects like managing different programming languages and importing community‐provided processes are often neglected, although they are crucial to meet the programming habits of the served communities . Overall, there is no system satisfying all the requirements reported in Section 1; nevertheless, examples can be given of flexible systems that may potentially meet them.…”

Realising Virtual Research Environments by Hybrid Data Infrastructures: the D4Science Experience

“…It offers a set of off-the-shelf algorithms including clustering algorithms such as DBScan. Moreover, it enables a simple integration and execution of user-defined algorithms expressed in a number of programming and scripting languages including R [32]. It currently embeds more than 100 different algorithms ranging from Anomalies Detection, Classification, Clustering, Simulation, Training, Bayesian Methods, Trends, and many more [33].…”

“…Virtual Research Environments by Hybrid Data Infrastructures the set of data sources integrated in the D4Science infrastructure; and (iii) the development of new algorithms and approaches aiming at enlarging the offering of the Statistical Manager service. Thanks to the openness of the gCube system, some of these developments can be performed by the community in the large, e.g., every scientists owning an algorithm worth to share can decide to integrate it into the Statistical Manager and benefit from a boost in performances [32].…”