Indexing multi-dimensional data in a cloud system

Wang, Jinbao; Wu, Sai; Gao, Hong; Li, Jianzhong; Ooi, Beng Chin

doi:10.1145/1807167.1807232

Cited by 144 publications

(98 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, these method only supports 1-dimensional indices which are insufficient for OLAP queries. There have been efforts to build distributed multi-dimensional indices on Cloud platforms based on R-trees or related multi-dimensional tree structures, such as [30], [31], [32]. However, these method do not support dimension hierarchies which are essential for OLAP queries.…”

Section: Related Workmentioning

confidence: 99%

A distributed tree data structure for real-time OLAP on cloud architectures

Dehne

Kong

Rau-Chaplin

et al. 2013

2013 IEEE International Conference on Big Data

View full text Add to dashboard Cite

Abstract-In contrast to queries for on-line transaction processing (OLTP) systems that typically access only a small portion of a database, OLAP queries may need to aggregate large portions of a database which often leads to performance issues. In this paper we introduce CR-OLAP, a Cloud based Real-time OLAP system based on a new distributed index structure for OLAP, the distributed PDCR tree, that utilizes a cloud infrastructure consisting of (m + 1) multi-core processors. With increasing database size, CR-OLAP dynamically increases m to maintain performance. Our distributed PDCR tree data structure supports multiple dimension hierarchies and efficient query processing on the elaborate dimension hierarchies which are so central to OLAP systems. It is particularly efficient for complex OLAP queries that need to aggregate large portions of the data warehouse, such as "report the total sales in all stores located in California and New York during the months February-May of all years". We evaluated CR-OLAP on the Amazon EC2 cloud, using the TPC-DS benchmark data set. The tests demonstrate that CR-OLAP scales well with increasing number of processors, even for complex queries. For example, on an Amazon EC2 cloud instance with eight processors, for a TPC-DS OLAP query stream on a data warehouse with 80 million tuples where every OLAP query aggregates more than 50% of the database, CR-OLAP achieved a query latency of 0.3 seconds which can be considered a real time response.

show abstract

Section: Related Workmentioning

confidence: 99%

A distributed tree data structure for real-time OLAP on cloud architectures

Dehne

Kong

Rau-Chaplin

et al. 2013

2013 IEEE International Conference on Big Data

View full text Add to dashboard Cite

show abstract

“…RT-CAN [8], QT-Chord [3], EMINC [10] and A-Tree [7] are four types of multi ple-dimensional indexes suitable for cloud data management systems. RT-CAN integrates CAN based routing protocol and the R-tree based indexing scheme to support efficient multi-dimensional query processing in a Cloud system.…”

Section: Query Optimization Through Indexesmentioning

confidence: 99%

Fast Multi-fields Query Processing in Bigtable Based Cloud Systems

Wang

Meng

2013

Web-Age Information Management

View full text Add to dashboard Cite

Abstract.With the rapid increase of data sizes, enterprise applications are migrating their backend data management and analytic systems into cloud based data management systems.Bigtable is among one of the major data models used by cloud storage systems as their storage layer. Such systems provide high scalability and schema flexibility, and support efficient point and range based queries based on rowkeys. However, Bigtable based systems have limited support on non-rowkey based queries and multiple-fields based queries, due to much overhead on invoking extra scanning of data. In this paper, we develop a system TNBGR(Telecom Network Browsing Gateway Records) on managing and querying large scale telecommunication data. TNBGR is built on top of HBase and MapReduce, with a focus on optimizing multi-fields query processing. TNBGR provides a novel application and system resource aware data allocation strategy to minimize data access through multi-layer region partitioning, resource parameterization, and balanced region distribution.The query composition dynamically updates application parameters based on tracked system statistics and automatically translates queries for MapReduce. Through additional query optimization by improving region locality, TNBGR achieves high efficiency on supporting multi-field queries. The experimental results show that our solution improves the performance of the queries by about 5 and 18 times respectively.

show abstract

“…As an example, Figure 1 shows the spatial distribution of users during two time interval [1,6] and [7,14]. For range query, find the users who are in the spatial range marked by the dashed line rectangle within time period [1,6], apparently, {u 1 , u 3 } is the result.…”

Section: Introductionmentioning

confidence: 99%

“…For range query, find the users who are in the spatial range marked by the dashed line rectangle within time period [1,6], apparently, {u 1 , u 3 } is the result. For 1NN query, if we want to find the users who are nearest to p 1 during time period [1,6] and [7,14], respectively, the result is u 2 for [1, 6] and u 1 for [7,14]. For GNN query, if we want to find the user who are nearest to p 1 and p 2 by summing the distances during time period [1,6], the result is u 2 .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Efficient Historical Query in HBase for Spatio-Temporal Decision Support

Chen

Zhang

et al. 2016

INT J COMPUT COMMUN

View full text Add to dashboard Cite

Comparing to last decade, technologies to gather spatio-temporal data are more and more developed and easy to use or deploy, thus tens of billions, even trillions of sensed data are accumulated, which poses a challenge to spatio-temporal Decision Support System (stDSS). Traditional database hardly supports such huge volume, and tends to bring performance bottleneck to the analysis platform. Hence in this paper, we argue to use NoSQL database, HBase, to replace traditional back-end storage system. Under such context, the well-studied spatio-temporal querying techniques in traditional database should be shifted to HBase system parallel. However, this problem is not solved well in HBase, as many previous works tackle the problem only by designing schema, i.e., designing row key and column key formation for HBase, which we don't believe is an effective solution. In this paper, we address this problem from nature level of HBase, and propose an index structure as a built-in component for HBase. STEHIX (Spatio-TEmporal Hbase IndeX) is adapted to two-level architecture of HBase and suitable for HBase to process spatio-temporal queries. It is composed of index in the meta table (the first level) and region index (the second level) for indexing inner structure of HBase regions. Base on this structure, three queries, range query, kNN query and GNN query are solved by proposing algorithms, respectively. For achieving load balancing and scalable kNN query, two optimizations are also presented. We implement STEHIX and conduct experiments on real dataset, and the results show our design outperforms a previous work in many aspects.

show abstract

Indexing multi-dimensional data in a cloud system

Cited by 144 publications

References 21 publications

A distributed tree data structure for real-time OLAP on cloud architectures

A distributed tree data structure for real-time OLAP on cloud architectures

Fast Multi-fields Query Processing in Bigtable Based Cloud Systems

Efficient Historical Query in HBase for Spatio-Temporal Decision Support

Contact Info

Product

Resources

About