Design and evaluation of decentralized online clustering

Quiroz, Andres; Parashar, Manish; Gnanasambandam, Nathan; Sharma, Naveen Kumar

doi:10.1145/2348832.2348837

Cited by 16 publications

(12 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, in this paper we adopt unsupervised learning, which does not require labelled training data and is used to find structures and patterns in data. For an extensive review on the these solutions we refer to [7] and, specifically on on-line clustering, to [8], [9]. Among the unsupervised solutions, we seek an algorithm that can process data fast, can handle mixed types, and ideally could process data in an online fashion.…”

Section: A Clustering As Unsupervised Machine Learningmentioning

confidence: 99%

Service Clustering for Autonomic Clouds Using Random Forest

Uriarte

Tsaftaris

Tiezzi

2015

2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

View full text Add to dashboard Cite

Abstract-Managing and optimising cloud services is one of the main challenges faced by industry and academia. A possible solution is resorting to self-management, as fostered by autonomic computing. However, the abstraction layer provided by cloud computing obfuscates several details of the provided services, which, in turn, hinders the effectiveness of autonomic managers. Data-driven approaches, particularly those relying on service clustering based on machine learning techniques, can assist the autonomic management and support decisions concerning, for example, the scheduling and deployment of services. One aspect that complicates this approach is that the information provided by the monitoring contains both continuous (e.g. CPU load) and categorical (e.g. VM instance type) data. Current approaches treat this problem in a heuristic fashion. This paper, instead, proposes an approach, which uses all kinds of data and learns in a data-driven fashion the similarities and resource usage patterns among the services. In particular, we use an unsupervised formulation of the Random Forest algorithm to calculate similarities and provide them as input to a clustering algorithm. For the sake of efficiency and meeting the dynamism requirement of autonomic clouds, our methodology consists of two steps: (i) off-line clustering and (ii) on-line prediction. Using datasets from real-world clouds, we demonstrate the superiority of our solution with respect to others and validate the accuracy of the on-line prediction. Moreover, to show the applicability of our approach, we devise a service scheduler that uses the notion of similarity among services and evaluate it in a cloud test-bed.

show abstract

Section: A Clustering As Unsupervised Machine Learningmentioning

confidence: 99%

Service Clustering for Autonomic Clouds Using Random Forest

Uriarte

Tsaftaris

Tiezzi

2015

2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

View full text Add to dashboard Cite

show abstract

“…DOC, introduced by Quiroz et al [6,13], was created to provide online and decentralized data analysis, using the collective computing resources in distributed systems. DOC is an online algorithm, thereby allowing short-term system behavior to be captured, as opposed to an offline approach that cannot capture short-term behavior.…”

Section: Decentralized Online Clustering (Doc)mentioning

confidence: 99%

“…(5) The resulting meta-clusters can now be used to identify these related clusters as a single cluster to be tracked. (6) The trajectory for each temporal cluster is updated by storing features in sorted order according to the analysis window in which they were found.…”

Section: Cluster Tracking Algorithmmentioning

confidence: 99%

“…Central to this framework is the scalable decentralized and online clustering (DOC) [6,7] and cluster tracking algorithm, which executes in-situ, i.e., on different cores, and in parallel with the simulation processes, and accesses simulation data directly and asynchronously (to the extent possible) via on-chip shared memory. The framework also provides programming support for composing in-situ "simulation plus analytics" workflows.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

In-situ feature-based objects tracking for data-intensive scientific and enterprise analytics workflows

et al. 2014

Self Cite

View full text Add to dashboard Cite

Emerging scientific simulations on leadership class systems are generating huge amounts of data and processing this data in an efficient and timely manner is critical for generating insights from the simulations. However, the increasing gap between computation and disk I/O speeds makes traditional data analytics pipelines based on post-processing cost prohibitive and often infeasible. In this paper, we investigate an alternate approach that aims to bring the analytics closer to the data using in-situ execution of data analysis operations. Specifically, we present the design, implementation and evaluation of a framework that can support in-situ feature-based objects tracking on distributed scientific datasets. Central to this framework is a scalable decentralized and online clustering, a cluster tracking algorithm, which executes in-situ (on different cores) in parallel with the simulation processes, and retrieves data from the simulations directly via on-chip shared memory. The results from our experimental evaluation demonstrate that the in-situ approach significantly reduces the cost of data movement, that the presented framework can support scalable featurebased objects tracking, and that it can be effectively used for in-situ analytics in large scale simulations.

show abstract

“…The algorithm specification, along with details about its implementation and robustness to failures, were the subject of previous publications [47]. Other applications of DOC have been also studied in the context of autonomic resource provisioning [45,58] and autonomic policy adaptation [46] Here, we explain the main characteristics of the algorithm, and refer the reader to the cited publications for further details.…”

Section: Enterprise Business Data Analyticsmentioning

confidence: 99%

Federating Advanced Cyberinfrastructures with Autonomic Capabilities

Diaz-Montes

Rodero

Zou

et al. 2014

Cloud Computing for Data-Intensive Applications

Self Cite

View full text Add to dashboard Cite

Cloud computing has emerged as a dominant paradigm that has been widely adopted by enterprises. Clouds provide on-demand access to computing utilities, an abstraction of unlimited computing resources, and support for ondemand scale up, scale down and scale out. Clouds are also rapidly joining high performance computing system, clusters and grids as viable platforms for scientific exploration and discovery. Furthermore, dynamically federated Cloud-ofClouds infrastructure can support heterogeneous and highly dynamic applications requirements by composing appropriate (public and/or private) cloud services and capabilities. As a result, providing scalable and robust mechanisms to federate distributed infrastructures and handle application workflows, that can effectively utilize them, is critical. In this chapter, we present a federation model to support the dynamic federation of resources and autonomic management mechanisms that coordinate multiple workflows to use resources based on objectives. We demonstrate the effectiveness of the proposed framework and autonomic mechanisms through the discussion of an experimental evaluation of illustrative use case application scenarios, and from these experiences, we discuss that such a federation model can support new types of application formulations.

show abstract

Design and evaluation of decentralized online clustering

Cited by 16 publications

References 21 publications

Service Clustering for Autonomic Clouds Using Random Forest

Service Clustering for Autonomic Clouds Using Random Forest

In-situ feature-based objects tracking for data-intensive scientific and enterprise analytics workflows

Federating Advanced Cyberinfrastructures with Autonomic Capabilities

Contact Info

Product

Resources

About