Automatic Generation of Workload Profiles Using Unsupervised Learning Pipelines

Buchaca, David; Berral, Josep Lluís; Carrera, David

doi:10.1109/tnsm.2017.2786047

Cited by 12 publications

(7 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The model can be used for descriptive, predictive and prescriptive analytics where the analytics output, for example, produces performance insight or predicts resource provisioning. The workload model can also be used for simulating workloads [77] and evaluating a system configuration [78]. Indeed, the workload-aware concept becomes a common aspect of different management architecture.…”

Section: Workload Modelingmentioning

confidence: 99%

Transfer Learning Based Performance Modeling And Effective Storage Management In Big Data Ecosystems

Alsayoud¹

2021

Preprint

View full text Add to dashboard Cite

Big data ecosystems contain a mix of sophisticated hardware storage components to support heterogeneous workloads. Storage components and the workloads interact and affect each other; therefore, their relationship has to consider when modeling workloads or managing storage. Efficient workload modeling guides optimal storage management decisions, and the right decisions help guarantee the workload’s needs. The first part of this thesis focuses on workload modeling efficiency, and the second part focuses on cost-effective storage management.<div>Workload performance modeling is an essential step in management decisions. The standard modeling approach constructs the model based on a historical dataset collected from one set of setups (scenario). The standard modeling approach requires the model to be reconstructed from scratch with every time the setups changes. To address this issue, we propose a cross-scenario modeling approach that improves the workload’s performance classification accuracy by up to 78% through adopting the Transfer Learning (TL).<br></div><div>The storage system is the most crucial component of the big data ecosystem, where the workload’s execution process starts by fetching data from it and ends by storing data into it. Thus, the workload’s performance is directly affected by storage capability. To provide a high I/O performance in the ecosystems, Solid State Drive (SSD) are utilized as a tier or as a cache on big data distributed ecosystems. SSDs have a short lifespan that is affected by data size and the number of writing operations. Balancing performance requirements and SSD’s lifespan consumption is never easy, and it’s even harder when interacting with a huge amount of data and with heterogeneous I/O patterns. In this thesis, we analysis big data workloads I/O pattern impacts on SSD’s lifespan when SSD is used as a tier or as a cache. Then, we design a Hidden Markov Model (HMM) based I/O pattern controller that manages workload placement and guarantees cost-effective storage that enhances the workload performance by up to 60%, and improves SSD’s lifespan by up to 40%. </div><div>The designed transfer learning modeling approach and the storage management solutions improve workload modeling accuracy, and the quality of the storage management policies while the testing setup changes.<br></div>

show abstract

Section: Workload Modelingmentioning

confidence: 99%

Transfer Learning Based Performance Modeling And Effective Storage Management In Big Data Ecosystems

Alsayoud¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Alam et al [24] 2018 Aral & Ovatman [27] 2018 Atrey et al [29] 2018 -Continued on next page- [34] 2018 Barrameda & Samaan [36] 2018 Borjigin et al [39] 2018 Bouet & Conan [40] 2018 Cheng et al [44] 2018 Diaz-Montes et al [53] 2018 Gill et al [60] 2018 Govindaraj & Artemenko [62] 2018 Guo & Shenoy [63] 2018 Guo et al [64] 2018 Guo et al [65] 2018 Hauser & Wesner [68] 2018 Heidari & Buyya [69] 2018 Jia et al [77] 2018 Jia et al [78] 2018 Khabbaz & Assi [83] 2018 Lahmann et al [88] 2018 Lin et al [93] 2018 Mikavica et al [109] 2018 Nawrocki & Sniezynski [117] 2018 Prakash et al [124] 2018 Prats et al [125] 2018 Rahimi et al [127] 2018 Sahni & Vidyarthi [131] 2018 Santos et al [133] 2018 Scheuner & Leitner [136] 2018 Simonis [139] 2018 Sofia & GaneshKumar [135] 2018 Stoyanov & Kollingbaum [141] 2018 Takahashi et al [142] 2018 Tesfatsion et al [144] 2018 Trihinas et al [145] 2018 Wang & Gelenbe [148] 2018 Wei et al [151] 2018 Xie & Jia [156] 2018 Yao & Ansari [160] 2018 Zhang & Wen [170] 2018 Zhang et al…”

Section: Cloud Resource Management: State Of the Artmentioning

confidence: 99%

“…Existing container orchestration tools exist for the deployment and management of containers, but these are still relatively young and still lack some important features that are offered in VM environments, for example for achieving high availability which includes live migration of running applications. [68] 2018 Prats et al [125] 2018 Scheuner & Leitner [136] 2018 Trihinas et al [145] 2018…”

Section: Highlights For Global Provisioning and Schedulingmentioning

confidence: 99%

“…Based on the obtained workload profiles, the estimated cost is calculated, which could help cloud users for finding alternate infrastructures that afford lower hosting costs while offering equal or better performance. Somewhat related, Prats et al introduced an approach for the automatic generation of workload profiles [125]. The authors examined and modeled application behavior by finding phases of similar behavior in the workloads.…”

Section: Application Demand Profilingmentioning

confidence: 99%

See 1 more Smart Citation

Resource Management in a Containerized Cloud: Status and Challenges

et al. 2019

View full text Add to dashboard Cite

Cloud computing heavily relies on virtualization, as with cloud computing virtual resources are typically leased to the consumer, for example as virtual machines. Efficient management of these virtual resources is of great importance, as it has a direct impact on both the scalability and the operational costs of the cloud environment. Recently, containers are gaining popularity as virtualization technology, due to the minimal overhead compared to traditional virtual machines and the offered portability. Traditional resource management strategies however are typically designed for the allocation and migration of virtual machines, so the question arises how these strategies can be adapted for the management of a containerized cloud. Apart from this, the cloud is also no longer limited to the centrally hosted data center infrastructure. New deployment models have gained maturity, such as fog and mobile edge computing, bringing the cloud closer to the end user. These models could also benefit from container technology, as the newly introduced devices often have limited hardware resources. In this survey, we provide an overview of the current state of the art regarding resource management within the broad sense of cloud computing, complementary to existing surveys in literature. We investigate how research is adapting to the recent evolutions within the cloud, being the adoption of container technology and the introduction of the fog computing conceptual model. Furthermore, we identify several challenges and possible opportunities for future research.

show abstract

“…The proposed methodology focuses on using CRBMs to boost clustering and prediction algorithms, to improve the quality of features like ship main engine power, navigation status and ship category, from each ships navigation traces. The decision of using CRBMs is based on their capacity to deal with AIS as multi-dimensional time-series [12], also encouraged by the methodology proposed by Buchaca et al [13] used for detecting phase behavior patterns on time-series. The CRBMs are used to extract and cluster temporal patterns, also to expand features from the time series, allowing non-time-aware predictors better accuracy.…”

Section: Introductionmentioning

confidence: 99%

Improving maritime traffic emission estimations on missing data with CRBMs

Gutierrez-Torre

Berral

Buchaca

et al. 2020

Engineering Applications of Artificial Intelligence

Self Cite

View full text Add to dashboard Cite

Maritime traffic emissions are a major concern to governments as they heavily impact the Air Quality in coastal cities. Ships use the Automatic Identification System (AIS) to continuously report position and speed among other features, and therefore this data is suitable to be used to estimate emissions, if it is combined with engine data. However, important ship features are often inaccurate or missing. State-of-the-art complex systems, like CALIOPE at the Barcelona Supercomputing Center, are used to model Air Quality. These systems can benefit from AIS based emission models as they are very precise in positioning the pollution. Unfortunately, these models are sensitive to missing or corrupted data, and therefore they need data curation techniques to significantly improve the estimation accuracy. In this work, we propose a methodology for treating ship data using Conditional Restricted Boltzmann Machines (CRBMs) plus machine learning methods to improve the quality of data passed to emission models that can also be applied to other GPS and time-series problems. Results show that we can improve the default methods proposed to cover missing data. In our results, we observed that using our method the models boosted their accuracy to detect otherwise undetectable emissions. In particular, we used a real data-set of AIS data, provided by the Spanish Port Authority, to estimate that thanks to our method, the model was able to detect 45% of additional emissions, representing 152 tonnes of pollutants per week in Barcelona and propose new features that may enhance emission modeling.

show abstract

Automatic Generation of Workload Profiles Using Unsupervised Learning Pipelines

Cited by 12 publications

References 30 publications

Transfer Learning Based Performance Modeling And Effective Storage Management In Big Data Ecosystems

Transfer Learning Based Performance Modeling And Effective Storage Management In Big Data Ecosystems

Resource Management in a Containerized Cloud: Status and Challenges

Improving maritime traffic emission estimations on missing data with CRBMs

Contact Info

Product

Resources

About