Transparent runtime parallelization of the R scripting language

SCBI_MapReduce, a New Ruby Task-Farm Skeleton for Automated Parallelisation and Distribution in Chunks of Sequences: The Implementation of a Boosted Blast+

“…That could explain why SCBI MapReduce skeleton shows a speed-up of 31-fold for 32 cores and 59-fold for 64 cores, even with sequence data (Figure 2(a)). This performance is better than the one displayed by the R package pR, where 32 cores provide speedups of 20-27-fold, depending on the process [25]. Several design reasons can also be invoked to explain such an efficiency [34]: (i) disk I/O operations are reduced to minimum (data are read only at the beginning and results are saved only at the end); (ii) absence of asymmetry impact (Figure 1(b)); (iii) the manager overhead is limited when using more than 2 cores and chunks of sequences (Tables 2 and 3); and (iv) longer tasks increased the efficiency because the manager is on standby most of the time, while waiting for the workers to finish, avoiding relaunching of internal or external programs for brief executions.…”

“…Parallelisation libraries for R language, besides Rmpi, are SPRINT [24] and pR [25] packages, whose their main advantage is that they require very little modification to the existing sequential R scripts and no expertise in parallel computing; however, the master worker suffers from communication overhead, and the authors recognise that their approach may not yield the optimal schedule [25]. Other parallelisation libraries are snow and nws that provide coordination and parallel execution facilities.…”

“…This paper describes SCBI MapReduce, a new task-farm skeleton for the Ruby scripting language [32] that gathers the requirements presented in the Introduction, and simplifies the creation of parallel and distributed software to researchers without skills in distributed programming. Even if customisation could appear more complicated than using existing libraries for parallelisation-such as OpenMP [18], BOINC [23], or R libraries such as Rmpi or SPRINT [24]-it is as simple as the parallelisation of R code using pR [25]. In contrast to these libraries, SCBI MapReduce is not constrained only to parallelisation (not allowing distribution nor grid computing) and is able to extract the complete distribution capabilities of any computer system.…”

Automatic Parallelization of Array-oriented Programs for a Multi-core Machine

“…Neither approach involves compiler manipulation, thus differing from our approach. The recent work of Li et al [16] on the scripting array language R is also on parallelizing its run-time routines but it did use sophisticated compiler technology to do this.…”

Cumulon

“…Targeting graph mining, PEGASUS [15] implements generalized iterated matrix-vector multiply efficiently on Hadoop. RIOT [24] and RevoScaleR focus on making statistical computing workloads in R I/O-efficient; pR [16] automatically parallelizes function calls and loops in R. Pig [17], Hive [20], and SciHadoop [4] are examples of higher-level languages and execution plan generators for MapReduce systems. Our work goes beyond these systems by addressing important usability issues of automatic hardware provisioning and configuration.…”