E3: an Elastic Execution Engine for Scalable Data Processing

Abstract

With the unprecedented growth of data generated by mankind nowadays, it has become critical to develop efficient techniques for processing these massive data sets. To tackle such challenges, analytical data processing systems must be extremely efficient, scalable, and flexible as well as economically effective. Recently, Hadoop, an open-source implementation of MapReduce, has gained interests as a promising big data processing system. Although Hadoop offers the desired flexibility and scalability, its performance has been noted to be suboptimal when it is used to process complex analytical tasks. This paper presents E 3 , an elastic and efficient execution engine for scalable data processing. E 3 adopts a "middle" approach between MapReduce and Dryad in that E 3 has a simpler communication model than Dryad yet it can support multi-stages job better than MapReduce. E 3 avoids reprocessing intermediate results by adopting a stage-based evaluation strategy and collocating data and user-defined (map or reduce) functions into independent processing units for parallel execution. Furthermore, E 3 supports block-level indexes, and built-in functions for specifying and optimizing data processing flows. Benchmarking on an in-house cluster shows that E 3 achieves significantly better performance than Hadoop, or put it another way, building an elastically scalable and efficient data processing system is possible.

Contextual crowd intelligence

“…The next layer (which is the security layer) enables users to protect data privacy by encryption. The third layer (which is the distributed processing layer) provides a distributed processing infrastructure called E 3 [9] that supports different parallel processing logics such as MapReduce [11], Directed Acyclic Graph (DAG) and SQL. The top layer (which is the analytics layer) exploits the contextual crowd intelligence for big data analytics.…”

Federation in Cloud Data Management: Challenges and Opportunities

“…The epiC system also introduces a trusted data service for users to protect data privacy by encryption and a subsystem E 3 [35] to manage distributed data processing. E 3 has been implemented based on the Actor concurrent programming model and designed for large clusters.…”

Database research at the National University of Singapore

“…The epiC system adopts an extensible design. The core of epiC provides two services: virtual block service (called VBS) which manages the cloud storage devices and a coordination framework (called E 3 [9]) which coordinates independent computations over federated systems. To analyze the data, users invoke a set of computing units (called Actors).…”