Towards zero-overhead static and adaptive indexing in Hadoop

Abstract

Several research works have focused on supporting index access in MapReduce systems. These works have allowed users to significantly speed up selective MapReduce jobs by orders of magnitude. However, all these proposals require users to create indexes upfront, which might be a difficult task in certain applications (such as in scientific and social applications) where workloads are evolving or hard to predict. To overcome this problem, we propose LIAH (Lazy Indexing and Adaptivity in Hadoop), a parallel, adaptive approach for indexing at minimal costs for MapReduce systems. The main idea of LIAH is to automatically and incrementally adapt to users' workloads by creating clustered indexes on HDFS data blocks as a byproduct of executing MapReduce jobs. Besides distributing indexing efforts over multiple computing nodes, LIAH also parallelises indexing with both map tasks computation and disk I/O. All this without any additional data copy in main memory and with minimal synchronisation. The beauty of LIAH is that it piggybacks index creation on map tasks, which read relevant data from disk to main memory anyways. Hence, LIAH does not introduce any additional read I/O-costs and exploit free CPU cycles. As a result and in contrast to existing adaptive indexing works, LIAH has a very low (or invisible) indexing overhead, usually for the very first job. Still, LIAH can quickly converge to a complete index, i.e. all HDFS data blocks are indexed. Especially, LIAH can trade early job runtime improvements with fast complete index convergence. We compare LIAH with HAIL, a state-of-the-art indexing technique, as well as with standard Hadoop with respect to indexing overhead and workload performance. In terms of indexing overhead, LIAH can completely index a dataset as a byproduct of only four MapReduce jobs while incurring a low overhead of 11% over HAIL for the very first MapReduce job only. In terms of workload performance, our results show that LIAH outperforms Hadoop by up to a factor of 52 and HAIL by up to a factor of 24.

Efficient storage, retrieval and analysis of poker hands: An adaptive data framework

“…There are many optimisation techniques that allow increasing the performance of MapReduce based solutions (Jiang et al, 2010;Thusoo et al, 2010). For instance, one way to improve Hadoop's performance is to design an optimised storage model that utilises various indexing solutions (Richter et al, 2014). Although the use of column oriented storage formats (RCFile, 2016;ORC, 2016) dominates the design of modern Hadoop-based data warehousing solutions, we present a storage model that is both read optimised and row based.…”

“…Primarily, they have been exploited by MapReduce jobs, and in the case of HAIL required decorating map tasks with customised annotations, specifying the selection predicates and the list of projected fields (Richter et al, 2014). Unfortunately, no indexing enhancements to the MapReduce interface are visible to the applications that conform to the original interface such as Hive or Spark.…”

On the analysis of big data indexing execution strategies

“…Lazy Indexing (LIAH) is proposed by Richter, Quiané-Ruiz [18] as adaptive indexing using clustered approach. LIAH uses offer rate to minimize indexing I/O cost and creates as many indexes as suggested by incoming queries.…”

“…Nevertheless to completely index all data blocks, low offer rate will require more MapReduce jobs. Due to this fact, LIAH has to compromise either indexing overhead or number of MapReduce jobs which provides motivation towards dynamically adapting offer rate [19]. Although query workload prediction is not required and unlike static indexing there is no replication factor dependency to consider number of index attributes in both of these approaches yet performing full scan for each new query and replicating data block for each new index attribute are the performance bottlenecks of LIAH.…”

“…To improve the efficiency of search and data retrieval process for voluminous data records many solutions have been proposed by researchers. For example, vertical partitioning [12], clustered attribute based indexing [6,9] for distributed parallel processing systems and clustered adaptive indexing [18] for changing query workload. Likewise in medical research, large distributed image data sets face the problem of multi-query optimization and a batch processing based image retrieval system [25] contributes in scheduling multiple query requests and minimized response time is achieved.…”