Modern Column Stores for Big Data Processing

Sridhar, K. T.

doi:10.1007/978-3-319-72413-3_8

Cited by 5 publications

(6 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ACID-compliant databases that conceptually represent data in the relational model, and that can be queried using SQL (Bach & Werner, 2016;Dhindsa, 2012), although challenged by the rise of various newer technologies such as NoSQL (Strauch, 2011) and NewSQL (Pavlo & Aslett, 2016), have often been found to be apt technology for a variety of workloads (Sridhar, 2017;Mike Stonebraker et al, 2005). Table 1 illustrates a simple data model (Dhindsa, 2012).…”

Section: Dbms Architecturesmentioning

confidence: 99%

“…Table 1 illustrates a simple data model (Dhindsa, 2012). A DBMS's physical storage is typically a one-dimensional structure of pages having a pre-defined size, which defines the granularity of I/O operations (Sridhar, 2017).…”

Section: Dbms Architecturesmentioning

confidence: 99%

“…With this type of physical storage organisation, a request for the value of a column X within a row N results in a specific operation (Dhindsa, 2012;Ordonez & Bellatreche, 2018;Sridhar, 2017) that:…”

Section: Row-storementioning

confidence: 99%

“…In contrast to a rowstore RDBMS, a columnar RDBMS vertically partitions a table, physically storing each column in a separate page. Each page contains values organised in an order that reflects the order of the rows (Sridhar, 2017). Figure 3 illustrates the same data model referred to in Table 1, as stored by a columnar RDBMS (Dhindsa, 2012).…”

Section: Column-storementioning

confidence: 99%

See 3 more Smart Citations

HTAP With Reactive Streaming ETL

Camilleri

Vella

Nezval

2021

Journal of Cases on Information Technology

View full text Add to dashboard Cite

In database management systems (DBMSs), query workloads can be classified as online transactional processing (OLTP) or online analytical processing (OLAP). These often run within separate DBMSs. In hybrid transactional and analytical processing (HTAP), both workloads may execute within the same DBMS. This article shows that it is possible to run separate OLTP and OLAP DBMSs, and still support timely business decisions from analytical queries running off fresh transactional data. Several setups to manage OLTP and OLAP workloads are analysed. Then, benchmarks on two industry standard DBMSs empirically show that, under an OLTP workload, a row-store DBMS sustains a 1000 times higher throughput than a columnar DBMS, whilst OLAP queries are more than 4 times faster on a columnar DBMS. Finally, a reactive streaming ETL pipeline is implemented which connects these two DBMSs. Separate benchmarks show that OLTP events can be streamed to an OLAP database within a few seconds.

show abstract

Section: Dbms Architecturesmentioning

confidence: 99%

Section: Dbms Architecturesmentioning

confidence: 99%

Section: Row-storementioning

confidence: 99%

Section: Column-storementioning

confidence: 99%

See 2 more Smart Citations

HTAP With Reactive Streaming ETL

Camilleri

Vella

Nezval

2021

Journal of Cases on Information Technology

View full text Add to dashboard Cite

show abstract

“…E veryday companies are generating large volumes of data and their traditional relational database management systems (a.k.a. row-stores) fail to handle this data efficiently [1]. An alternative to row-stores are column-stores which store information about a logical entity as separate columns in multiple locations on the disk [2].…”

Section: Introductionmentioning

confidence: 99%

Distributed ATrie Group Join: Towards Zero Network Cost

2020

View full text Add to dashboard Cite

The combination of powerful parallel frameworks and on-demand commodity hardware in distributed computing has made both analytics and decision support systems canonical to enterprises of all sizes. The unprecedented volumes of data stacked by companies present challenges to process analytical queries efficiently. This data is often organised as star schema, in which star join and group-by are ubiquitous and expensive operations. Although parallel frameworks such as Apache Spark facilitate join and groupby, the implementation can only process two tables at a time and fail to handle the excessive network communication, disk spills and multiple scans of data. In this paper, we present Distributed ATrie Group Join (DATGJ), a fast distributed star join and group-by algorithm for column-stores. DATGJ uses divide and broadcast-based joining technique where the fact table columns are partitioned equally and fast hash table (FHT) for each dimension table are broadcasted. This technique helps it avoid cross communication between workers and disk spills. DATGJ performs a single scan of partitioned fact table columns and use FHT to join data. FHT uses Robin Hood hashing with the upper limit on number of probes and achieve significant speed up during join. DATGJ performs group-by and aggregation leveraging progressive materialisation and realising grouping attributes as a tree shaped deterministic finite automation known as Aggregate Trie or ATrie. We evaluated our algorithm using Star Schema Benchmark (SSBM) to show that it is 1.5X to 6X faster than the most prominent approaches while having zero data shuffle and consistently perform well with addition of resources and in memory-constrained scenarios. INDEX TERMS big data processing, column-stores, distributed processing, parallel group-by, paralleljoin.

show abstract