Dynamic data transformation for low latency querying in big data systems

Abstract

Abstract-Big data storage technologies inherently entail high latency characteristics, preventing users from performing efficient ad-hoc querying and interactive visualization on large and distributed datasets. Most of the existing approaches addressing this issue thrive on de-normalization of the static data schema and creation of application specific (i.e. hardcoded) materialized views, which certainly reduce data access latency but at the expense of flexibility. In this regard, this paper proposes an approach that relies on an iterative process of data transformation intended to generate read-optimized data schemas. The transformation process is able to automatically identify optimization opportunities (e.g. materialized views, missing indexes), by analyzing the original data schema and the record of queries issued by users and client applications against the data set. An experimental evaluation of the proposed approach evidences a significant reduction in the query latency, ranging from 81.60% to 99.99%.

A Workload-Driven Approach for View Selection in Large Dimensional Datasets

“…In this sense, this paper delves further into the approach introduced by Ordonez et al [7], particularly by elaborating on an automatic mechanism for materialized view selection and creation. The mechanism presented in the following sections relies also on syntactic analysis of query workloads issued against a dimensionally modeled data collection.…”

“…It is noteworthy that the approach in [7] was concerned with defining a data transformation framework on a conceptual level, while the mechanism discussed herein addresses an actual realization of such framework, tackling the problem of materialized view selection on large data collections. Likewise, the approach introduced by Vanhove et al [8] -that served as inspiration for the framework proposed in [7]-is not particularly concerned with reducing query latency, but with enabling live data migration between different data storage technologies, irrespective of the query-workload.…”

“…It is noteworthy that the approach in [7] was concerned with defining a data transformation framework on a conceptual level, while the mechanism discussed herein addresses an actual realization of such framework, tackling the problem of materialized view selection on large data collections. Likewise, the approach introduced by Vanhove et al [8] -that served as inspiration for the framework proposed in [7]-is not particularly concerned with reducing query latency, but with enabling live data migration between different data storage technologies, irrespective of the query-workload. In this sense, the contribution of the proposed mechanism lies in three key features: (i ) a vector representation that encodes not only the query-attribute usage, but also the basic structure of analytical queries, enabling a more precise and also regular representation of the query set, (ii ) a measure of query distance tightly suited to the structure of the formulated feature vector representation providing a more accurate method for estimating query relatedness, instead of plain Hamming distance used in existing approaches, and (iii ) a scalable procedure for candidate view generation that relies on a measure of cluster consistency, which in turn uses the above-mentioned query dissimilarity metric to unambiguously identify materializable groups of related queries.…”

“…A key difference between these two types of workloads lies in the data models and structures they operate on: OLTP systems work on top of highly normalized data models, while OLAP workloads run against denormalized schemas featuring precomputed views derived from transactional business data. Results of a previous experimental study [7] evidence that using such read-optimized structures alone is not enough for analytical processing applications to meet strict response time requirements, even for small datasets.…”

“…Based on the work of Vanhove et al, a framework that serves as conceptual foundation for the mechanism this paper reports on is presented in [7]. The intuition behind that framework was to progressively optimize the schema of a base dataset by applying a sequence of data transformation operations (e.g.…”