The BigDAWG Polystore System

Abstract

This paper presents a new view of federated databases to address the growing need for managing information that spans multiple data models. This trend is fueled by the proliferation of storage engines and query languages based on the observation that "no one size fits all". To address this shift, we propose a polystore architecture; it is designed to unify querying over multiple data models. We consider the challenges and opportunities associated with polystores. Open questions in this space revolve around query optimization and the assignment of objects to storage engines. We introduce our approach to these topics and discuss our prototype in the context of the Intel Science and Technology Center for Big Data.

From conceptual design to performance optimization of ETL workflows: current state of research and open problems

“…A rapidly growing need for analyzing Big Data calls for novel architectures for warehousing the data, such as a data lake [57] or a polystore [58]. In both of the architectures, ETL processes serve similar purposes as in traditional DW architectures.…”

Exploring complex and big data

“…A query engine needs to support additional functionalities, including means for (i) identifying relevant data sets for answering a given query, i.e., to understand the content of the data lake, (ii) discovering formats of data sets, (iii) converting data on-the-fly to the format preferred by a user (Duggan et al, 2015;Liu and Wang, 2016), and (iv) appropriately visualizing query results (Chen and Zhang, 2014). To this end, appropriate and rich metadata are needed.…”

“…Polystore. In the work of Duggan et al (2015), yet another alternative big data integration architecture was proposed, called a polystore. In this architecture, data sets are organized into the so-called islands of information.…”

Fast queries over heterogeneous data through engine customization

“…These polystore systems initially combined Hadoop with an RDBMS [3,22]. Newer proposals [17,23] bundle more engines to better fit more use cases. To treat multiple engines as one, the overall solution uses middleware to perform cross-system query optimization, query splitting, data exchange between systems, etc.…”

“…Data types such as hierarchies, however, are not a natural fit for tables. Another alternative is the data federation of heterogeneous data sources [17,23]. The dominant approach in this case is packaging together multiple query engines, using the appropriate one for each specialized scenario, and relying on a middleware layer to integrate data from different sources.…”