A containerized analytics framework for data and compute-intensive pipeline applications

Abstract

The joint effort of scientific collaborations and the expanding data market creates demand for high-performance and dataintensive analytics infrastructures that can exploit the potential of heterogeneous multi-core architectures with dynamic and scalable execution environments. Contemporary approaches focus on developing efficient parallel application models, but lack the flexibility of efficiently integrating and utilizing native or accelerator-based code. In this work, we illustrate a novel approach on mending this shortcoming and offering seamless application integration into a highly versatile execution infrastructure. The centerpiece is a framework of containerized execution units and management thereof for satisfying the diverse requirements of data analytics pipelines and its stages. Containers not only ease distribution and deployment of applications, but, more importantly enable an efficient synthesis of different stage implementation variants aimed towards exploiting heterogeneous computing resources. Consequently, this approach allows the infrastructure to utilize mainstream data and compute-intensive techniques and paradigms to achieve the goal of efficient pipeline execution. We present our approach in form of a requirement analysis, a multi-tier architecture description, and deployment scenarios based on our current prototype implementation.

Incorporating Data Context to Cost-Effectively Automate End-to-End Data Wrangling

“…However, Data Civilizer does not benefit from the systematic use of data context that is described here; it could be extended to do so. On the contrary, Big data analytics platforms such as [7] and [6] focus on optimising the execution of composable data anlytics workflows according to data locality and data flow. Our platform focuses on the design and implementation of a scalable and modularised data wrangling workflow in a domain-independent manner.…”

“…Such steps can be carried out using traditional Extract-Transform-Load (ETL) or Big Data analytics platforms [5], [6], [7], both requiring significant manual involvement in specifying, configuring, programming or tuning many of the steps [8], [9]. It is widely reported that intense manual involvement in such processes is expensive (e.g., [10]), often representing more than half the time of data scientists.…”

Data context informed data wrangling

“…Such steps can be carried out using Extract-Transform-Load (ETL) [3] or Big Data analytics platforms [4], both necessitating significant manual involvement in specifying, configuring, programming or tuning many of the steps. It is widely reported that intense manual involvement in such processes is expensive (e.g.…”

“…Transformed, integrated and repaired records schematic correspondences. We utilize the Coma 3.0 community edition4 , specifically, the Coma workflow (configuration 7001) combining different metadata-based match heuristic. When data context is provided in D, each such data set is used as a partial extensional representation of the target to carry out instance based matching with the source (line 5).…”