Flexible and efficient IR using array databases

Abstract

The Matrix Framework is a recent proposal by Information Retrieval (IR) researchers to flexibly represent information retrieval models and concepts in a single multidimensional array framework. We provide computational support for exactly this framework with the array database system SRAM (Sparse Relational Array Mapping), that works on top of a DBMS. Information retrieval models can be specified in its comprehension-based array query language, in a way that directly corresponds to the underlying mathematical formulas. SRAM efficiently stores sparse arrays in (compressed) relational tables and translates and optimizes array queries into relational queries. In this work, we describe a number of array query optimization rules. To demonstrate their effect on text retrieval, we apply them in the TREC TeraByte track (TREC-TB) efficiency task, using the Okapi BM25 model as our example. It turns out that these optimization rules enable SRAM to automatically translate the BM25 array queries into the relational equivalent of inverted list processing including compression, score materialization and quantization, such as employed by custom-built IR systems. The use of the high-performance MonetDB/X100 relational backend, that provides transparent database compression, allows the system to achieve very fast response times with good precision and low resource usage.

SciQL, a query language for science applications

“…Maier et al [24] injected "a call to order" into the database community, in which the authors stated that the key problem for the relational DBMSs to support scientific applications is the lack of support for ordered data structures, like multidimensional arrays and time series. The call has been well accepted by the community, considering the various proposals on DBMS support (e.g., [4,6,9,16,20,31]), SQL language extensions (e.g., [3,21,27]) and algebraic frameworks (e.g., [9,22,25]) for ordered data.…”

“…In the area of building/extending DBMS with array support, the systems can be divided into three groups: i) simulating arrays on top of a relational DBMS, such as RAM [36] and SRAM [9]; ii) storing arrays as BLOBs in relational DBMSs and using an array executor to deal with array specific queries, such as RasDaMan [4]; and iii) enhancing a relational DBMS with array as a primary data type and providing native support for array oriented operations, such as SciQL and SciDB [6,34,10].…”

“…Nevertheless, they have not been too successful in the science domain beyond the management of meta data and workflow status. This mismatch between application needs and database technology has long been recognized [6,34,15,18,9,20,16,4,31]. In particular, an efficient implementation of array and time series concepts is missing [15,34,24,1].…”

The Data Cyclotron query processing scheme

“…MonetDB is a modern fully functional column-store database system [7,29,10]. It stores data column-wise in binary structures, called Binary Association Tables, or BATs, which represent a mapping from an OID to a base type value.…”

Emerging Database Systems in Support of Scientific Data

“… Arrays. The Sparse Relational Array Mapping (SRAM) project maps large (scientific) array-based data-sets into MonetDB BATs, and offers a high-level comprehension-based query language [CHZ+08]. This language is subsequently optimized on various levels before being translated into BAT Algebra .…”