Scalable and dynamically balanced shared-everything OLTP with physiological partitioning

Tözün, Pinar; Pandis, Ippokratis; Johnson, Ryan; Ailamaki, Anastasia

doi:10.1007/s00778-012-0278-6

Cited by 19 publications

(14 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The scenario turns out to be an optimisation problem and mixed-integer linear programming (MILP) solver is used to find an optimal answer. By logically partitioning the physical data accesses, [16] propose physiological partitioning (PLP)-a transaction processing approach where physical access to database pages and indexes are continuously repartitioned and maintain load-balance based on data access patterns. However, applications that have less impact on the underlying storages can be penalised as mentioned by the authors, and understanding the directional flow and dependencies of data access for each transactions can be a costly operation.…”

Section: Related Workmentioning

confidence: 99%

Workload-aware incremental repartitioning of shared-nothing distributed databases for scalable OLTP applications

Kamal

Murshed

Buyya

2016

Future Generation Computer Systems

View full text Add to dashboard Cite

h i g h l i g h t s• We propose incremental repartitioning of distributed OLTP databases for high-scalability.• We model two incremental repartitioning algorithm and lookup mechanism.• We develop a unique transaction generation model for simulation.• We derive novel impact metrics for distributed transactions.• Simulation results indicate adaptability of the methods scalable OLTP applications. On-line Transaction Processing (OLTP) applications often rely on shared-nothing distributed databases that can sustain rapid growth in data volume. Distributed transactions (DTs) that involve data tuples from multiple geo-distributed servers can adversely impact the performance of such databases, especially when the transactions are short-lived and these require immediate responses. The k-way min-cut graph clustering based database repartitioning algorithms can be used to reduce the number of DTs with acceptable level of load balancing. Web applications, where DT profile changes over time due to dynamically varying workload patterns, frequent database repartitioning is needed to keep up with the change. This paper addresses this emerging challenge by introducing incremental repartitioning. In each repartitioning cycle, DT profile is learnt online and k-way min-cut clustering algorithm is applied on a special sub-graph representing all DTs as well as those non-DTs that have at least one tuple in a DT. The latter ensures that the min-cut algorithm minimally reintroduces new DTs from the non-DTs while maximally transforming existing DTs into non-DTs in the new partitioning. Potential load imbalance risk is mitigated by applying the graph clustering algorithm on the finer logical partitions instead of the servers and relying on random one-to-one cluster-to-partition mapping that naturally balances out loads. Inter-server datamigration due to repartitioning is kept in check with two special mappings favouring the current partition of majority tuples in a cluster-the many-to-one version minimising data migrations alone and the one-toone version reducing data migration without affecting load balancing. A distributed data lookup process, inspired by the roaming protocol in mobile networks, is introduced to efficiently handle data migration without affecting scalability. The effectiveness of the proposed framework is evaluated on realistic TPC-C workloads comprehensively using graph, hypergraph, and compressed hypergraph representations used in the literature. To compare the performance of any incremental repartitioning framework without any bias of the external min-cut algorithm due to graph size variations, a transaction generation model is developed that can maintain a target number of unique transactions in any arbitrary observation window, irrespective of new transaction arrival rate. The overall impact of DTs at any instance is estimated from the exponential moving average of the recurrence period of unique transactions to avoid transient fluctuations. The effectiveness and adaptability of the proposed incremental repartition...

show abstract

Section: Related Workmentioning

confidence: 99%

Workload-aware incremental repartitioning of shared-nothing distributed databases for scalable OLTP applications

Kamal

Murshed

Buyya

2016

Future Generation Computer Systems

View full text Add to dashboard Cite

show abstract

“…Whenever multiple instances must collectively process a request, shared-nothing databases require expensive distributed consensus protocols, such as two-phase commit, which many argue are inherently non-scalable [12,24]. Similarly, handling data and access skew is problematic [61].…”

Section: Shared-nothing Database Deploymentsmentioning

confidence: 99%

“…Practitioners report that even commercial shared-everything systems with support for nonuniform memory architectures (NUMA) are hard to tune for modern servers [25,14,70]. On the other hand, sharednothing deployments [57] face the challenges of (a) higher execution costs when distributed transactions are required [12,16,24,50], even within a single node, particularly if the communication occurs between slower links (e.g., across CPU sockets); and (b) load imbalances due to skew [61].…”

Section: Introductionmentioning

confidence: 99%

Characterization of the Impact of Hardware Islands on OLTP

Porobic

Pandis

Branco³

et al. 2015

The VLDB Journal

Self Cite

View full text Add to dashboard Cite

Modern hardware is abundantly parallel and increasingly heterogeneous. The numerous processing cores have non-uniform access latencies to the main memory and processor caches, which causes variability in the communication costs. Unfortunately, database systems mostly assume that all processing cores are the same and that microarchitecture differences are not significant enough to appear in critical database execution paths. As we demonstrate in this paper, however, non-uniform core topology does appear in the critical path and conventional database architectures achieve suboptimal and even worse, unpredictable performance. We perform a detailed performance analysis of OLTP deployments in servers with multiple cores per CPU (multicore) and multiple CPUs per server (multisocket). We compare different database deployment strategies where we vary the number and size of independent database instances running on a single server, from a single shared-everything instance to fine-grained shared-nothing configurations. We quantify the impact of non-uniform hardware on various deployments by (a) examining how efficiently each deployment uses the available hardware resources and (b) measuring the impact of distributed transactions and skewed requests on different workloads. We show that no strategy is optimal for all cases and that the best choice depends on the combination of hardware topology and workload characteristics. Finally, we argue that transaction processing systems must be aware of the hardware topology in order to achieve predictably high performance.

show abstract

“…On the other hand, one of the recent proposals for adaptive repartitioning algorithms targets physiologically partitioned shared-everything systems [18]. The load on each partition is monitored using histograms and work queues.…”

Section: B Data Partitioning Strategiesmentioning

confidence: 99%

ATraPos: Adaptive transaction processing on hardware Islands

Porobic

Liarou

Tözün

et al. 2014

2014 IEEE 30th International Conference on Data Engineering

Self Cite

View full text Add to dashboard Cite

Abstract-Nowadays, high-performance transaction processing applications increasingly run on multisocket multicore servers. Such architectures exhibit non-uniform memory access latency as well as non-uniform thread communication costs. Unfortunately, traditional shared-everything database management systems are designed for uniform inter-core communication speeds. This causes unpredictable access latencies in the critical path. While lack of data locality may be a minor nuisance on systems with fewer than 4 processors, it becomes a serious scalability limitation on larger systems due to accesses to centralized data structures.In this paper, we propose ATraPos, a storage manager design that is aware of the non-uniform access latencies of multisocket systems. ATraPos achieves good data locality by carefully partitioning the data as well as internal data structures (e.g., state information) to the available processors and by assigning threads to specific partitions. Furthermore, ATraPos dynamically adapts to the workload characteristics, i.e., when the workload changes, ATraPos detects the change and automatically revises the data partitioning and thread placement to fit the current access patterns and hardware topology.We prototype ATraPos on top of an open-source storage manager Shore-MT and we present a detailed experimental analysis with both synthetic and standard (TPC-C and TATP) benchmarks. We show that ATraPos exhibits performance improvements of a factor ranging from 1.4 to 6.7x for a wide collection of transactional workloads. In addition, we show that the adaptive monitoring and partitioning scheme of ATraPos poses a negligible cost, while it allows the system to dynamically and gracefully adapt when the workload changes.

show abstract

Scalable and dynamically balanced shared-everything OLTP with physiological partitioning

Cited by 19 publications

References 41 publications

Workload-aware incremental repartitioning of shared-nothing distributed databases for scalable OLTP applications

Workload-aware incremental repartitioning of shared-nothing distributed databases for scalable OLTP applications

Characterization of the Impact of Hardware Islands on OLTP

ATraPos: Adaptive transaction processing on hardware Islands

Contact Info

Product

Resources

About