Packed-Memory Quadtree: A cache-oblivious data structure for visual exploration of streaming spatiotemporal big data

Abstract

The visual analysis of large multidimensional spatiotemporal datasets poses challenging questions regarding storage requirements and query performance. Several data structures have recently been proposed to address these problems that rely on indexes that pre-compute different aggregations from a known-a-priori dataset. Consider now the problem of handling streaming datasets, in which data arrive as one or more continuous data streams. Such datasets introduce challenges to the data structure, which now has to support dynamic updates (insertions/deletions) and rebalancing operations to perform selfreorganizations. In this work, we present the Packed-Memory Quadtree (PMQ), a novel data structure designed to support visual exploration of streaming spatiotemporal datasets. PMQ is cache-oblivious to perform well under different cache configurations. We store streaming data in an internal index that keeps a spatiotemporal ordering over the data following a quadtree representation, with support for real-time insertions and deletions. We validate our data structure under different dynamic scenarios and compare to competing strategies. We demonstrate how PMQ could be used to answer different types of visual spatiotemporal range queries of streaming datasets.

Online List Labeling: Breaking the $\log^2n$ Barrier

“…Khayyat, Lucia, Singh, Ouzzani, Papotti, Quiané-Ruiz, Tang and Kalnis [42] apply it to handle dynamic database updates in their inequality join algorithms. Toss, Pahins, Raffin and Comba [68] present a packed memory quadtree, which supports large streaming spatiotemporal datasets. De Leo and Boncz [45] present the rewired memory array, an implementation of a packed memory array which improves on its practical performance.…”

Packed Memory Arrays - Rewired

“…Rather, the elements can be moved in batch of updates. A similar approach has been considered in the recent paper [31], where sparse arrays are considered to store and process temporal streams of tweets. In [30], the PMA is utilised to store a graph on GPU following the compressed storage row (CSR) representation.…”

“…• We present a new bulk loading algorithm. It is particularly suitable for the streaming scenario [30], [31], where the cardinality of the array is kept constant, and updates, featuring the same amount of insertions and deletions, are executed in batches at regular intervals. This paper is organised as follows: In Section II, we summarise PMAs and their properties.…”

Fast Concurrent Reads and Updates with PMAs

“…Both our treatment and implementation are based on the variant Rewired Memory Array (RMA) [9]. Some practical applications showcasing the usage of PMAs have been published so far [11,16,29,31]. Still, their implementations have been found simpler and less performing than standard B + -trees or the RMA [9].…”