Scalable script-based data analysis workflows on clouds

Marozzo, Fabrizio; Talia, Domenico; Trunfio, Paolo

doi:10.1145/2534248.2534261

Cited by 11 publications

(7 citation statements)

References 19 publications

(23 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…JavaScript for Cloud is a JavaScript-based language for programming data analysis workflows. It has been introduced as the script-based language for the DMCF [21]. The web interface of DMCF allows to design and execute workflows programmed by the JS4Cloud language, by providing an environment similar to that used to develop visual workflows in the same framework.…”

Section: The Javascript For Cloud Languagementioning

confidence: 99%

“…Then, k unlabeled datasets are specified as input, with k D 4 (line 17). Each of the k input datasets is classified by n predictors using the n models generated by J48 and by m predictors using the m models generated by JRip; therefore, for each of the k input datasets, n C m classified datasets are generated (lines [18][19][20][21][22]. As a final step, k weighted voters are executed; the i-th voter receives the n C m classified datasets generated from the i-th input and the n C m models and returns the final classified dataset for the i-th input (lines [25][26].…”

Section: Ensemble Learning Workflowmentioning

confidence: 99%

See 1 more Smart Citation

JS4Cloud: script‐based workflow programming for scalable data analysis on cloud platforms

Marozzo

Talia

Trunfio

2015

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

SUMMARYWorkflows are an effective paradigm to model complex data analysis processes, such as knowledge discovery in databases applications, which can be efficiently executed on distributed computing systems such as a Cloud platform. Data analysis workflows can be designed through visual programming, which is a convenient design approach for high-level users. On the other hand, script-based workflows are a useful alternative to visual workflows, because they allow expert users to program complex applications more effectively. In order to provide Cloud users with an effective script-based data analysis workflow formalism, we designed the JS4Cloud language. The main benefits of JS4Cloud are as follows: (i) it extends the well-known JavaScript language while using only its basic functions (arrays, functions, and loops); (ii) it implements both a data-driven task parallelism that automatically spawns ready-to-run tasks to the Cloud resources and data parallelism through an array-based formalism; and (iii) these two types of parallelism are exploited implicitly so that workflows can be programmed in a fully sequential way, which frees users from duties like work partitioning, synchronization, and communication. We describe how JS4Cloud has been integrated within the data mining cloud framework (DMCF), a system supporting the scalable execution of data analysis workflows on Cloud platforms. In particular, we describe how data analysis workflows modeled as JS4Cloud scripts are processed by DMCF by exploiting parallelism to enable their scalable execution on Clouds. Finally, we present some data analysis workflows developed with JS4Cloud and the performance results obtained by executing such workflows on DMCF.

show abstract

Section: The Javascript For Cloud Languagementioning

confidence: 99%

Section: Ensemble Learning Workflowmentioning

confidence: 99%

JS4Cloud: script‐based workflow programming for scalable data analysis on cloud platforms

Marozzo

Talia

Trunfio

2015

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“…La salida de la validación del modelo fue la siguiente, como se observa en la tabla 8, con una primera columna que es la instancia; la segunda no se tiene en cuenta porque todos los atributos fueron marcados con un "? ", por ende la columna actual puede ser ignorada, se limita a establecer que cada clase pertenece a una clase desconocida; la columna predicted muestra la predicción de cada instancia, y la columna error prediction refleja la probabilidad de que la instancia en realidad pertenezca a la clase (Marozzo, Talia, & Trunfio, 2013).…”

Section: Validación Del Modelounclassified

Minería de datos aplicada a la demanda del transporte aéreo en Ocaña, Norte de Santander

Gómez

Ibáñez

2015

View full text Add to dashboard Cite

Este artículo muestra la aplicación de la minería de datos para predecir la demanda del servicio aéreo en Ocaña, Norte de Santander, respecto a los pares origen-destino Ocaña-Bogotá, Ocaña-Bucaramanga, Ocaña-Medellín, Ocaña-Cúcuta, Ocaña-Barranquilla; se utilizan datos de estudios realizados para estimar la demanda de un nuevo medio de transporte en la ciudad. Esta investigación sigue las fases del proceso de extracción del conocimiento en bases de datos. En la etapa de minería de datos se seleccionó como técnica los árboles de decisión y como algoritmo el J48. Con la aplicación de este algoritmo se encontró que las personas que viajan mensual y semanalmente eligen el avión, igual que si viajan por motivos de salud; si trabajan y se dirigen a Barranquilla, Bogotá y Medellín, eligen la buseta.

show abstract

“…Several workflows commonly require the execution of multiple data-intensive operations as loading, transformation, and aggregation ( Mattoso et al, 2010 ). Multiple computational paradigms can be used for the design and execution of workflows, e.g., shell and Python scripts ( Marozzo, Talia & Trunfio, 2013 ), Big Data frameworks (e.g., Hadoop and Spark) ( Guedes et al, 2020b ), but they are usually managed by complex engines named Workflow Management Systems (WfMS). A key feature that a WfMS must address is the efficient and automatic management of parallel processing activities in High Performance Computing (HPC) environments ( Ogasawara et al, 2011 ).…”

Section: Introductionmentioning

confidence: 99%

Provenance-and machine learning-based recommendation of parameter values in scientific workflows

Silva

Pacitti

Paes

et al. 2021

PeerJ Computer Science

View full text Add to dashboard Cite

Scientific Workflows (SWfs) have revolutionized how scientists in various domains of science conduct their experiments. The management of SWfs is performed by complex tools that provide support for workflow composition, monitoring, execution, capturing, and storage of the data generated during execution. In some cases, they also provide components to ease the visualization and analysis of the generated data. During the workflow’s composition phase, programs must be selected to perform the activities defined in the workflow specification. These programs often require additional parameters that serve to adjust the program’s behavior according to the experiment’s goals. Consequently, workflows commonly have many parameters to be manually configured, encompassing even more than one hundred in many cases. Wrongly parameters’ values choosing can lead to crash workflows executions or provide undesired results. As the execution of data- and compute-intensive workflows is commonly performed in a high-performance computing environment e.g., (a cluster, a supercomputer, or a public cloud), an unsuccessful execution configures a waste of time and resources. In this article, we present FReeP—Feature Recommender from Preferences, a parameter value recommendation method that is designed to suggest values for workflow parameters, taking into account past user preferences. FReeP is based on Machine Learning techniques, particularly in Preference Learning. FReeP is composed of three algorithms, where two of them aim at recommending the value for one parameter at a time, and the third makes recommendations for n parameters at once. The experimental results obtained with provenance data from two broadly used workflows showed FReeP usefulness in the recommendation of values for one parameter. Furthermore, the results indicate the potential of FReeP to recommend values for n parameters in scientific workflows.

show abstract

Scalable script-based data analysis workflows on clouds

Cited by 11 publications

References 19 publications

JS4Cloud: script‐based workflow programming for scalable data analysis on cloud platforms

JS4Cloud: script‐based workflow programming for scalable data analysis on cloud platforms

Minería de datos aplicada a la demanda del transporte aéreo en Ocaña, Norte de Santander

Provenance-and machine learning-based recommendation of parameter values in scientific workflows

Contact Info

Product

Resources

About