Programming knowledge discovery workflows in service‐oriented distributed systems

Abstract

In several scientific and business domains, very large data repositories are generated. To find interesting and useful information in those repositories, efficient data mining techniques and knowledge discovery processes must be used. The exploitation of data mining techniques in science helps scientists in hypothesis formation and gives them a support on their scientific practices, whereas in industrial processes, data mining can exploit existing data sources as a real value for companies that can take advantage from the knowledge that can be extracted from their large data sources. Data mining tasks are often composed by multiple stages that may be linked to each other to form various execution flows. Moreover, data mining tasks are often distributed because they involve data and tools located over geographically distributed environments. Therefore, it is fundamental to exploit effective paradigms, such as services and workflows, to model data mining tasks that are both multi-staged and distributed. This paper discusses data mining services and workflows for analyzing scientific data in high-performance distributed environments such as Grids and Clouds. We discuss how it is possible to define basic and complex services for supporting distributed data mining tasks in Grids. We also present a workflow formalism and a service-oriented programming framework, named DIS3GNO, for designing and running distributed knowledge discovery processes in the Knowledge Grid system. DIS3GNO supports all the phases of a knowledge discovery process, including composition, execution, and results visualization. After introducing DIS3GNO, some relevant use cases implemented by it and a performance evaluation of the system are discussed. Abstract resource Concrete resource Location set KNOWLEDGE DISCOVERY WORKFLOWS IN DISTRIBUTED SYSTEMS 1499 Figure 17. Parameter sweeping workflow.581,012 instances and is stored in a file having a size of 72 MB. From this dataset, we extracted three datasets with 72,500, 145,000, and 290,000 instances and a file size of 9 MB, 18 MB, and 36 MB, respectively. Then, we used DIS3GNO to perform a classification analysis on each of those datasets.DIS3GNO has been used to run an application in which eight independent instances of the J48 algorithm perform a different classification task on the covertype dataset. In particular, each J48 instance has been asked to classify data using a different value of confidence, ranging from 0.15 to 0.50. The same application has been executed using a number of computing nodes ranging from one to eight to evaluate the system speedup.The workflow corresponding to the application is shown in Figure 17. It includes a dataset node (representing the covertype dataset) connected to eight tool nodes, each one associated with an instance of the J48 classification algorithm with a different value of confidence (ranging from 0.15 to 0.50). These nodes are in turn connected to another tool node, associated with a model chooser that selects the best classification model among those le...

Scalable script-based data analysis workflows on clouds

“…DMCF supports a large variety of data mining processes, including single-task applications, parameter sweeping application, and workflow-based applications. Following the approach proposed in [2], DMCF represents knowledge discovery workflows as graphs whose nodes denote resources (datasets, data mining tools, data mining models) and whose edges denote dependencies among resources. A Web-based user interface allows users to compose workflows and to submit them for execution to Cloud platforms, following a Software-as-a-Service approach.…”

JS4Cloud: script‐based workflow programming for scalable data analysis on cloud platforms

“…DMCF supports a large variety of processing patterns that can be used in data mining including single-task applications, parameter-sweeping applications, and workflow-based Copyright JS4Cloud JavaScript Implicit data-driven task paral-Low No Yes lelism and data parallelism applications. Following the approach proposed in [18], DMCF represents knowledge discovery workflows as graphs whose nodes denote resources (datasets, data analysis tools, and mining models) and whose edges denote dependencies among resources. A web-based user interface allows users to compose their applications and to submit them for execution to the Cloud platform, following a SaaS approach.…”

“…Then, k unlabeled datasets are specified as input, with k D 4 (line 17). Each of the k input datasets is classified by n predictors using the n models generated by J48 and by m predictors using the m models generated by JRip; therefore, for each of the k input datasets, n C m classified datasets are generated (lines [18][19][20][21][22]. As a final step, k weighted voters are executed; the i-th voter receives the n C m classified datasets generated from the i-th input and the n C m models and returns the final classified dataset for the i-th input (lines [25][26].…”

Cloud4SNP

“…In the following we outline the Data Mining Cloud Framework and explain how Cloud4SNP has been implemented on it. Following the approach proposed in [19] and [4], the Data Mining Cloud Framework models knowledge discovery workflows as graphs whose nodes represent resources (datasets, data mining tools, data mining models) and whose edges represent dependencies between resources. The framework includes a Website to compose workflows and to submit their execution to the Cloud, following a Software-as-a-Service approach.…”

“…www.affymetrix.com4 In the rest of the paper the terms SNPs and alleles are used interchangeably 5. DMET-Analyzer is available at: http://sourceforge.net/projects/dmetanalyzer/files.…”