The Buffer Tree: A Technique for Designing Batched External Data Structures

Abstract

Abstract. We present a technique for designing external memory data structures that support batched operations I/O efficiently. We show how the technique can be used to develop external versions of a search tree, a priority queue, and a segment tree, and give examples of how these structures can be used to develop I/Oefficient algorithms. The developed algorithms are either extremely simple or straightforward generalizations of known internal memory algorithms-given the developed external data structures.Key Words. I/O efficiency, Internal memory algorithms, Batched external data structures, Buffer tree. Introduction.In recent years, increasing attention has been given to Input/Outputefficient (or I/O-efficient) algorithms. This is due to the fact that communication between fast internal memory and slower external memory such as disks is the bottleneck in many computations involving massive datasets. The significance of this bottleneck is increasing as internal computation gets faster and parallel computing gains popularity.Much work has been done on designing external versions of data structures designed for internal memory. Most of these structures are designed to be used in on-line settings, where queries should be answered immediately and within a good worst case number of I/Os. As a consequence they often do not take advantage of the available main memory, leading to suboptimal performance when they are used in solutions for batched (or offline) problems. Therefore several techniques have been developed for solving massive batched problems without using external data structures.In this paper we present a technique for designing external data structures that take advantage of the large main memory. We do so by only requiring good amortized performance and by allowing query operations to be batched. We also show how the developed data structures can be used in simple and I/O-efficient algorithms for a number of fundamental computational geometry and graph problems. Our technique has subsequently been used in the development of a large number of I/O-efficient algorithms in these and several other problem areas.

An Efficient Indexing Structure for Multidimensional Categorical Range Aggregation Query

“…Starting with B-Tree, people have already used cache to temporarily store pages in memory, which greatly reduces I/O. The Buffer-Tree [15,16] is the first index structure to have a separate cache on each node, which collecting small-scale written operation and the contents of cache will be written back to the node when the cache space of node is full. But doing this does not dynamic enough, it requires large cache space, and few other nodes are rarely written.…”

Operação de carga-rápida (bulk-loading) em métodos de acesso métricos

“…As operações de carga-rápida baseadas em buffer foram desenvolvidas através de técnicas que utilizamáreas de armazenamento temporário para evitar o alto custo de acesso a disco. Uma técnica bastante conhecida na literaturaé a Buffer-tree [Arge, 2003]. De maneira sucinta, quando os dados não cabem mais em memória, essa técnica pode ser utilizada para manter a eficiência de algoritmos que funcionam apenas em memória principal.…”

“…Em adição aos métodos baseados em ordenação, existem duas outras formas para fazer a carga-rápida: baseando-se no uso de buffers e por amostragem dos dados. A primeira forma, também conhecida como buffer-based bulk loading, utiliza a técnica da Buffer-tree [Arge, 2003]. Essa técnica pode ser utilizada para manter a eficiência de algoritmos de otimização que funcionam apenas em memória principal.…”

Scalable generation of random graphs

“…• Pushing and removing n elements from a priority queue is possible with sort(n) I/Os [23,22], and may be even cheaper for special cases such as integers of bounded magnitude. This allows Time Forward Processing:…”

“…determinisitc ids of new nodes ⇒ recursion stops copied from pseudorandom entry ⇒ recursion continues O(sort(m)). The other one is based on weighted sampling using a tailor-made variant of a Bu er Tree [23,22] and yields the same I/O complexity. The second generator is parallelized by processing subtrees individually.…”