Workload-driven adaptive data partitioning and distribution — The Cumulus approach

Fetai, Ilir; Murezzan, Damian; Schuldt, Heiko

doi:10.1109/bigdata.2015.7363940

Cited by 10 publications

(14 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fetai, Murezzan & Schuldt [ 56 ] presented Cumulus, an adaptive data fragmentation approach that can identify characteristic access patterns of transaction mixes, determine data partitions based on these patterns, and dynamically re-fragment data if access patterns change. The approach evaluation was performed with the TPC-C benchmark and it was shown that Cumulus significantly increased overall system performance in an OLTP configuration compared to static data partitioning approaches.…”

Section: Description Of the Set Of Work By Categorymentioning

confidence: 99%

Application of Dynamic Fragmentation Methods in Multimedia Databases: A Review

Castro-Medina

Rodríguez-Mazahua

López-Chau

et al. 2020

Entropy

View full text Add to dashboard Cite

Fragmentation is a design technique widely used in multimedia databases, because it produces substantial benefits in reducing response times, causing lower execution costs in each operation performed. Multimedia databases include data whose main characteristic is their large size, therefore, database administrators face a challenge of great importance, since they must contemplate the different qualities of non-trivial data. These databases over time undergo changes in their access patterns. Different fragmentation techniques presented in related studies show adequate workflows, however, some do not contemplate changes in access patterns. This paper aims to provide an in-depth review of the literature related to dynamic fragmentation of multimedia databases, to identify the main challenges, technologies employed, types of fragmentation used, and characteristics of the cost model. This review provides valuable information for database administrators by showing essential characteristics to perform proper fragmentation and to improve the performance of fragmentation schemes. The reduction of costs in fragmentation methods is one of the most desired main properties. To fulfill this objective, the works include cost models, covering different qualities. In this analysis, a set of characteristics used in the cost models of each work is presented to facilitate the creation of a new cost model including the most used qualities. In addition, different data sets or reference points used in the testing stage of each work analyzed are presented.

show abstract

Section: Description Of the Set Of Work By Categorymentioning

confidence: 99%

Application of Dynamic Fragmentation Methods in Multimedia Databases: A Review

Castro-Medina

Rodríguez-Mazahua

López-Chau

et al. 2020

Entropy

View full text Add to dashboard Cite

show abstract

“…In another similar work, Truyen et al have proposed K8-Scalar [52], an extensible workbench for evaluating different self-adaptive approaches to auto-scale NoSQL database clusters. Similarly, Fetai et al [53] presents Cumulus that minimizes distributed transactions through adaptive data repartitioning.…”

Section: Self-adaptive Systemsmentioning

confidence: 99%

SCOPE: self-adaptive and policy-based data management middleware for federated clouds

Rafique

Landuyt

Truyen

et al. 2019

J Internet Serv Appl

View full text Add to dashboard Cite

A federated cloud storage setup which integrates and utilizes storage resources from multiple cloud storage providers has become an increasingly popular and attractive paradigm for the persistence tier in cloud-based applications (e.g., SaaS applications, IoT applications, etc). However, federated cloud storage setups are prone to run-time dynamicity: many dynamic properties impact the way such a setup is governed and evolved over time, e.g., storage providers enter or leave the market; QoS metrics and SLA guarantees may change over time; etc. In general, existing federated cloud systems are oblivious to dynamic properties of the underlying operational environment, resulting in both sub-optimal data management decisions and costly SLA violations. Additionally, due to the sheer complexity of cloud-based applications coupled with the heterogeneous and volatile nature of federated cloud setups, the complexity of building, maintaining, and expending such applications increases dramatically and therefore managing them manually is no longer simply an option. To address these concerns, we present SCOPE, a policy-based and autonomic middleware that provides self-adaptiveness for data management in federated clouds. We have validated SCOPE in the context of a realistic SaaS application, performed an extensive functional validation, and conducted a thorough experimental evaluation. The evaluation results demonstrate (i) the ability of the middleware to perform data management decisions that take into account the run-time dynamicity (i.e., dynamic properties) of a federated cloud storage setup to meet the promised SLAs, and (ii) the self-adaptive behavior of SCOPE without the need for operator intervention. In addition, our in-depth performance evaluation results indicate that the benefits are achieved with acceptable performance overhead, and as such highlight the applicability of the proposed middleware for real-world application cases.

show abstract

“…Pavlo et al [19] developed an alternative approach that is also capable of stored procedure routing and replicated secondary indexes. Fetai et al focus especially on cloud environments [8].…”

Section: Related Workmentioning

confidence: 99%

Towards learning a partitioning advisor with deep reinforcement learning

Hilprecht

Binnig

Röhm

2019

Proceedings of the Second International Workshop on Exploiting Artificial Intelligence Techniques for Data Management

View full text Add to dashboard Cite

Commercial data analytics products such as Microsoft Azure SQL Data Warehouse or Amazon Redshift provide ready-touse scale-out database solutions for OLAP-style workloads in the cloud. While the provisioning of a database cluster is usually fully automated by cloud providers, customers typically still have to make important design decisions which were traditionally made by the database administrator such as selecting the partitioning schemes. In this paper we introduce a learned partitioning advisor for analytical OLAP-style workloads based on Deep Reinforcement Learning (DRL). The main idea is that a DRL agent learns its decisions based on experience by monitoring the rewards for different workloads and partitioning schemes. We evaluate our learned partitioning advisor in an experimental evaluation with different databases schemata and workloads of varying complexity. In the evaluation, we show that our advisor is not only able to find partitionings that outperform existing approaches for automated partitioning design but that it also can easily adjust to different deployments. This is especially important in cloud setups where customers can easily migrate their cluster to a new set of (virtual) machines.

show abstract

Workload-driven adaptive data partitioning and distribution — The Cumulus approach

Cited by 10 publications

References 22 publications

Application of Dynamic Fragmentation Methods in Multimedia Databases: A Review

Application of Dynamic Fragmentation Methods in Multimedia Databases: A Review

SCOPE: self-adaptive and policy-based data management middleware for federated clouds

Towards learning a partitioning advisor with deep reinforcement learning

Contact Info

Product

Resources

About