A spatial bitmap-based index for geographical data warehouses

Siqueira, Thiago Luís Lopes; Ciferri, Ricardo Rodrigues; Times, Valéria Cesário; Ciferri, Cristina Dutra de Aguiar

doi:10.1145/1529282.1529582

Cited by 12 publications

(24 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, it sets an offset variable to distinguish between the inequality operators with or without equality conditions (lines [13][14]. It then visits the values in L2 in the desired order, which is to sequentially scan the permutation array from left to right (lines [15][16][17][18][19][20][21][22]. For each tuple visited in L2, it first sets all bits for those t in T whose Y values are smaller than the Y value of the current tuple in T (lines [16][17][18], i.e., those tuples in T that satisfy the second join condition.…”

Section: Iejoinmentioning

confidence: 99%

“…For each tuple visited in L2, it first sets all bits for those t in T whose Y values are smaller than the Y value of the current tuple in T (lines [16][17][18], i.e., those tuples in T that satisfy the second join condition. It then uses the other offset array to find those tuples in T that also satisfy the first join condition (lines [19][20][21][22]. It finally returns all join results (line 23).…”

Section: Iejoinmentioning

confidence: 99%

“…The outer loop will take O(m · n) time (lines [15][16][17][18][19][20][21][22]. Hence, the total time complexity of the algorithm is O(m · log m + n · log n + m · n).…”

Section: Iejoinmentioning

confidence: 99%

“…We also extract metadata that contain the block ID, and the min/max values of each referenced attribute value from each data block (lines 8-15). Then, we create kMT 1 × kMT 2 virtual block combinations and filter out block combinations that do not generate results (lines [16][17][18][19]. Notice that blocks with non-intersecting min-max values do not produce results.…”

Section: Distributed Inequality Joinsmentioning

confidence: 99%

See 3 more Smart Citations

Lightning fast and space efficient inequality joins

Khayyat

Lucia²,

Singh³

et al. 2015

Proc. VLDB Endow.

View full text Add to dashboard Cite

Inequality joins, which join relational tables on inequality conditions, are used in various applications. While there have been a wide range of optimization methods for joins in database systems, from algorithms such as sort-merge join and band join, to various indices such as B + -tree, R * -tree and Bitmap, inequality joins have received little attention and queries containing such joins are usually very slow. In this paper, we introduce fast inequality join algorithms. We put columns to be joined in sorted arrays and we use permutation arrays to encode positions of tuples in one sorted array w.r.t. the other sorted array. In contrast to sort-merge join, we use space efficient bit-arrays that enable optimizations, such as Bloom filter indices, for fast computation of the join results. We have implemented a centralized version of these algorithms on top of PostgreSQL, and a distributed version on top of Spark SQL. We have compared against well known optimization techniques for inequality joins and show that our solution is more scalable and several orders of magnitude faster.1. ONCE UPON A TIME . . . Bob1 , a data analyst working for an international provider of cloud services, wanted to analyze revenue and utilization trends from different regions. In particular, he wanted to find out all those transactions from the West-Coast that last longer and produce smaller revenues than any transaction in the East-Coast. In other words, he was looking for any customer from the West-Coast who rented a virtual machine for more hours than any customer from the East-Coast, but who paid less. Figure 1 illustrates a data instance for both tables. He wrote the following join query for such a task: Figure 1: East-Coast and West-Coast transactions Bob first ran Qt over 200K transactions on the distributed system storing the data (System-X). Given that the input dataset is ∼1GB, he expected to have his answer in a minute or so. However, he waited for more than three hours without seeing any result. He immediately thought that this problem comes from System-X and killed the query. He then used an open-source DBMS-X to run his query. Although join is by far the most important and most studied operator in the relational algebra [1], Bob had to wait for over two hours until DBMS-X returned the results. He found that Qt is processed by DBMS-X as a Cartesian product followed by a selection predicate, which is problematic due to the huge number of unnecessary intermediate results.In the meantime, Bob heard that a big DBMS vendor was in town to highlight the power of their recently released distributed DBMS to process big data (DBMS-Y). So he visited them with a small (few KBs) dataset sample of the tables to run Qt. Surprisingly, DBMS-Y could not run Qt for even that small sample! He spent 45 minutes waiting while one of the DBMS-Y experts was trying to solve the issue. Bob left the query running and the vendor never contacted him again. In fact, DBMS-Y is using underneath the same open-source DBMS-X that Bob tried before. He t...

show abstract

Section: Iejoinmentioning

confidence: 99%

Section: Iejoinmentioning

confidence: 99%

“…The outer loop will take O(m · n) time (lines [15][16][17][18][19][20][21][22]. Hence, the total time complexity of the algorithm is O(m · log m + n · log n + m · n).…”

Section: Iejoinmentioning

confidence: 99%

Section: Distributed Inequality Joinsmentioning

confidence: 99%

See 2 more Smart Citations

Lightning fast and space efficient inequality joins

Khayyat

Lucia²,

Singh³

et al. 2015

Proc. VLDB Endow.

View full text Add to dashboard Cite

show abstract

“…Instead of search trees, FastBit relies on compressed bitmask indices [46,47] to provide efficient subselections and can significantly out-perform other approaches [37,40]. However, this class of data management techniques relies almost exclusively on extracting and aggregating pre-computed information.…”

Section: Related Workmentioning

confidence: 99%

Feature-Based Statistical Analysis of Combustion Simulation Data

Bennett

Krishnamoorthy

Liu

et al. 2011

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Fig. 1. Our framework provides a natural and intuitive work-flow for the exploration of global trends in feature-based statistics. By efficiently encoding hierarchical meta-data in a pre-processing step, interactive data exploration of the equivalent of one terabyte of simulation data is performed on a commodity desktop.Abstract-We present a new framework for feature-based statistical analysis of large-scale scientific data and demonstrate its effectiveness by analyzing features from Direct Numerical Simulations (DNS) of turbulent combustion. Turbulent flows are ubiquitous and account for transport and mixing processes in combustion, astrophysics, fusion, and climate modeling among other disciplines. They are also characterized by coherent structure or organized motion, i.e. nonlocal entities whose geometrical features can directly impact molecular mixing and reactive processes. While traditional multi-point statistics provide correlative information, they lack nonlocal structural information, and hence, fail to provide mechanistic causality information between organized fluid motion and mixing and reactive processes. Hence, it is of great interest to capture and track flow features and their statistics together with their correlation with relevant scalar quantities, e.g. temperature or species concentrations. In our approach we encode the set of all possible flow features by pre-computing merge trees augmented with attributes, such as statistical moments of various scalar fields, e.g. temperature, as well as length-scales computed via spectral analysis. The computation is performed in an efficient streaming manner in a pre-processing step and results in a collection of meta-data that is orders of magnitude smaller than the original simulation data. This meta-data is sufficient to support a fully flexible and interactive analysis of the features, allowing for arbitrary thresholds, providing per-feature statistics, and creating various global diagnostics such as Cumulative Density Functions (CDFs), histograms, or time-series. We combine the analysis with a rendering of the features in a linked-view browser that enables scientists to interactively explore, visualize, and analyze the equivalent of one terabyte of simulation data. We highlight the utility of this new framework for combustion science; however, it is applicable to many other science domains.

show abstract