External Data Access And Indexing In AsterixDB

Alamoudi, Abdullah A.; Grover, Raman; Carey, Michael J.; Borkar, Vinayak

doi:10.1145/2806416.2806428

Cited by 19 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since SQL-like queries of most big spatial data processing systems are not ANSI-standard and not eicient as spatial RDBMS, Sphinx achieves good performance over systems like SpatialHadoop. AsterixDB [6,10,11] incorporates LSM-based data storage and a set of indexing techniques including B + -tree and R-tree. It supports a complete query language, AQL, that uses Hyracks [45] as a query execution engine.…”

Section: Other Big Spatio-temporal Infrastructuresmentioning

confidence: 99%

A Survey on Spatio-temporal Data Analytics Systems

2022

View full text Add to dashboard Cite

Due to the surge of spatio-temporal data volume, the popularity of location-based services and applications, and the importance of extracted knowledge from spatio-temporal data to solve a wide range of real-world problems, a plethora of research and development work has been done in the area of spatial and spatio-temporal data analytics in the past decade. The main goal of existing works was to develop algorithms and technologies to capture, store, manage, analyze, and visualize spatial or spatio-temporal data. The researchers have contributed either by adding spatio-temporal support with existing systems, by developing a new system from scratch, or by implementing algorithms for processing spatio-temporal data. The existing ecosystem of spatial and spatio-temporal data analytics systems can be categorized into three groups, (1) spatial databases (SQL and NoSQL), (2) big spatial data processing infrastructures, and (3) programming languages and GIS software. Since existing surveys mostly investigated infrastructures for processing big spatial data, this survey has explored the whole ecosystem of spatial and spatio-temporal analytics. This survey also portrays the importance and future of spatial and spatio-temporal data analytics.

show abstract

Section: Other Big Spatio-temporal Infrastructuresmentioning

confidence: 99%

A Survey on Spatio-temporal Data Analytics Systems

2022

View full text Add to dashboard Cite

show abstract

“…However, one can run a query on existing data only, as data updates are not allowed in HDFS. AsterixDB [7,11,12] is a full-fledged big data management system, which incorporates LSM-based data storage and a set of indexing techniques including B + -tree and R-tree. It supports a complete query language, AQL, that uses Hyracks [36] as a query execution engine.…”

Section: Other Big Spatio-temporal Infrastructuresmentioning

confidence: 99%

“…These systems are implemented mainly by extending the MapReduce framework Hadoop [86], Spark [85,268,269], and NoSQL database systems [56,98] to incorporate spatial and temporal data types, partitioning and indexing techniques, geometric operations, and a SQL-like query language. However, a few of them have been developed either from scratch [27,28,154] or by extending systems other than Hadoop, Spark, and NoSQL databases [6,7,66]. Recently, Python libraries such as DASK [53] and RAPIDS [233] emerged as parallel and distributed platforms for processing big data.…”

Section: Introductionmentioning

confidence: 99%

A Survey on Spatio-temporal Data Analytics Systems

Alam¹,

Torgo²,

Bifet³

2021

Preprint

View full text Add to dashboard Cite

Due to the surge of spatio-temporal data volume, the popularity of location-based services and applications, and the importance of extracted knowledge from spatio-temporal data to solve a wide range of real-world problems, a plethora of research and development work has been done in the area of spatial and spatiotemporal data analytics in the past decade. The main goal of existing works was to develop algorithms and technologies to capture, store, manage, analyze, and visualize spatial or spatio-temporal data. The researchers have contributed either by adding spatio-temporal support with existing systems, by developing a new system from scratch for processing spatio-temporal data, or by implementing algorithms for mining spatio-temporal data. The existing ecosystem of spatial and spatio-temporal data analytics can be categorized into three groups, (1) spatial databases (SQL and NoSQL), ( 2) big spatio-temporal data processing infrastructures, and (3) programming languages and software tools for processing spatio-temporal data. Since existing surveys mostly investigated big data infrastructures for processing spatial data, this survey has explored the whole ecosystem of spatial and spatio-temporal analytics along with an up-to-date review of big spatial data processing systems. This survey also portrays the importance and future of spatial and spatio-temporal data analytics. CCS Concepts: • General and reference → Surveys and overviews; • Information systems → Spatialtemporal systems; Parallel and distributed DBMSs.

show abstract

“…The idea behind hybrid data solutions is to combine the best features of Hadoop and the RDBMS together (Alamoudi et al, 2015;Abouzeid et al, 2009). In our work, we try to bring many of the existing DB features and incorporate them into Hadoop without utilising database systems as auxiliary tools to support them.…”

Section: Related Workmentioning

confidence: 99%

Efficient storage, retrieval and analysis of poker hands: An adaptive data framework

Gorawski

Lorek

2017

International Journal of Applied Mathematics and Computer Science

View full text Add to dashboard Cite

In online gambling, poker hands are one of the most popular and fundamental units of the game state and can be considered objects comprising all the events that pertain to the single hand played. In a situation where tens of millions of poker hands are produced daily and need to be stored and analysed quickly, the use of relational databases no longer provides high scalability and performance stability. The purpose of this paper is to present an efficient way of storing and retrieving poker hands in a big data environment. We propose a new, read-optimised storage model that offers significant data access improvements over traditional database systems as well as the existing Hadoop file formats such as ORC, RCFile or SequenceFile. Through index-oriented partition elimination, our file format allows reducing the number of file splits that needs to be accessed, and improves query response time up to three orders of magnitude in comparison with other approaches. In addition, our file format supports a range of new indexing structures to facilitate fast row retrieval at a split level. Both index types operate independently of the Hive execution context and allow other big data computational frameworks such as MapReduce or Spark to benefit from the optimized data access path to the hand information. Moreover, we present a detailed analysis of our storage model and its supporting index structures, and how they are organised in the overall data framework. We also describe in detail how predicate based expression trees are used to build effective file-level execution plans. Our experimental tests conducted on a production cluster, holding nearly 40 billion hands which span over 4000 partitions, show that multi-way partition pruning outperforms other existing file formats, resulting in faster query execution times and better cluster utilisation.

show abstract

External Data Access And Indexing In AsterixDB

Cited by 19 publications

References 14 publications

A Survey on Spatio-temporal Data Analytics Systems

A Survey on Spatio-temporal Data Analytics Systems

A Survey on Spatio-temporal Data Analytics Systems

Efficient storage, retrieval and analysis of poker hands: An adaptive data framework

Contact Info

Product

Resources

About