Hadoop vs. Spark: Impact on Performance of the Hammer Query Engine for Open Data Corpora

Abstract

The Hammer prototype is a query engine for corpora of Open Data that provides users with the concept of blind querying. Since data sets published on Open Data portals are heterogeneous, users wishing to find out interesting data sets are blind: queries cannot be fully specified, as in the case of databases. Consequently, the query engine is responsible for rewriting and adapting the blind query to the actual data sets, by exploiting lexical and semantic similarity. The effectiveness of this approach was discussed in our previous works. In this paper, we report our experience in developing the query engine. In fact, in the very first version of the prototype, we realized that the implementation of the retrieval technique was too slow, even though corpora contained only a few thousands of data sets. We decided to adopt the Map-Reduce paradigm, in order to parallelize the query engine and improve performances. We passed through several versions of the query engine, either based on the Hadoop framework or on the Spark framework. Hadoop and Spark are two very popular frameworks for writing and executing parallel algorithms based on the Map-Reduce paradigm. In this paper, we present our study about the impact of adopting the Map-Reduce approach and its two most famous frameworks to parallelize the Hammer query engine; we discuss various implementations of the query engine, either obtained without significantly rewriting the algorithm or obtained by completely rewriting the algorithm by exploiting high level abstractions provided by Spark. The experimental campaign we performed shows the benefits provided by each studied solution, with the perspective of moving toward Big Data in the future. The lessons we learned are collected and synthesized into behavioral guidelines for developers approaching the problem of parallelizing algorithms by means of Map-Reduce frameworks.

J-CO: A Platform-Independent Framework for Managing Geo-Referenced JSON Data Sets

“…We will also investigate the possibility to integrate the J-CO-QL Engine with a Map-Reduce platform, such as Spark, that we successfully experimented for building a blind querying engine for Open Data sets [49] and for JSON data sets stored within JSON document stores [50]. In fact, in order to process actual Big Data, this solution appears to be promising; nonetheless, we will maintain the loosely-coupled approach, so as to keep user interfaces independent of computational resources actually adopted to process J-CO-QL queries.…”

Blind Queries Applied to JSON Document Stores

“…Hammer is the framework developed for blind querying Open Data portals, fully explained in [25] and briefly summarized in this section.…”

“…Note that the thresholds we used for defining the configurations are independent of the specific application domain. In fact, we used exactly the thresholds that in our previous works on Hammer [3,25] where considered to study the effectiveness of blind querying open data portals.…”

“…Spark is still a Map-Reduce framework, where Map-Reduce programming can be explicit or implicit. In Reference [25], we presented various implementation of the former Hammer framework for querying Open Data portals, that adopt both Hadoop and Spark; the results show that Spark can be very efficient, in particular if its high-level programming abstractions are exploited. Thus, since the Spark version demonstrated to be the most efficient one, the novel HammerJDB framework is based on the Spark version of the query-engine core.…”

“…Thus, since the Spark version demonstrated to be the most efficient one, the novel HammerJDB framework is based on the Spark version of the query-engine core. Hammer and HammerJDB are implemented on top of MongoDB, but they use it only as a pure storage technology and do not rely on its internal query language and (consequently) on the Map-Reduce support provided by it; this choice allows us to be independent of specific execution platforms and storage technology during the development; for example, we moved form the original Hadoop-based implementation to the current Spark-based implementation, seamlessly with respect to the storage technology; furthermore, the scalability of the approach (see Reference [25]) is guaranteed independently of the storage technology.…”