Recurring job optimization in scope

Motivated by streaming and data analytics scenarios where many queries operate on the same data and perform similar computations, we propose program consolidation for merging multiple user-defined functions (UDFs) that operate on the same input. Program consolidation exploits common computations between UDFs to generate an equivalent optimized function whose execution cost is often much smaller (and never greater) than the sum of the costs of executing each function individually. We present a sound consolidation calculus and an effective algorithm for consolidating multiple UDFs. Our approach is purely static and uses symbolic SMTbased techniques to identify shared or redundant computations. We have implemented the proposed technique on top of the Naiad data processing system. Our experiments show that our algorithm dramatically improves overall job completion time when executing user-defined filters that operate on the same data and perform similar computations.

show abstract

“…6 In total, our database consists of 31152 tweets in English, Spanish and Portuguese. The query families filter tweets by:…”

Section: Methodsmentioning

confidence: 99%

Consolidation of queries with user-defined functions

Sousa

Dillig

Vytiniotis

et al. 2014

Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation

View full text Add to dashboard Cite

show abstract

“…We make use of this assumption when choosing which samples to create. This assumption has been empirically observed in a variety of realworld production workloads [3,5] and is also true of the query trace we use for our primary evaluation (a 2-year query trace from Conviva Inc). We do not assume any prior knowledge of the specific values or predicates used in these clauses.…”

Section: Setting and Assumptionsmentioning

confidence: 95%

Blink and it's done

et al. 2012

Self Cite

View full text Add to dashboard Cite

In this demonstration, we present BlinkDB, a massively parallel, sampling-based approximate query processing framework for running interactive queries on large volumes of data. The key observation in BlinkDB is that one can make reasonable decisions in the absence of perfect answers. BlinkDB extends the Hive/HDFS stack and can handle the same set of SPJA (selection, projection, join and aggregate) queries as supported by these systems. BlinkDB provides real-time answers along with statistical error guarantees, and can scale to petabytes of data and thousands of machines in a fault-tolerant manner. Our experiments using the TPC-H benchmark and on an anonymized real-world video content distribution workload from Conviva Inc. show that BlinkDB can execute a wide range of queries up to 150× faster than Hive on MapReduce and 10 − 150× faster than Shark (Hive on Spark) over tens of terabytes of data stored across 100 machines, all with an error of 2 − 10%.

show abstract

“…Applications in a cluster are recurring [17]. This means that the res = { n ∈ neighbours | n.stagesCompleted = max(n.stagesCompleted) } 6:…”

Section: The Recommendation Workflowmentioning

confidence: 99%

Using machine learning to optimize parallelism in big data applications

Hernández

Pérez

Gupta

et al. 2018

Future Generation Computer Systems

View full text Add to dashboard Cite

In-memory cluster computing platforms have gained momentum in the last years, due to their ability to analyse big amounts of data in parallel. These platforms are complex and difficult-to-manage environments. In addition, there is a lack of tools to better understand and optimize such platforms that consequently form backbone of big data infrastructure and technologies. This directly leads to underutilization of available resources and application failures in such environment. One of the key aspects that can address this problem is optimization of the task parallelism of application in such environments. In this paper, we propose a machine learning based method that recommends optimal parameters for task parallelization in big data workloads. By monitoring and gathering metrics at system and application level, we are able to find statistical correlations that allow us to characterize and predict the effect of different parallelism settings on performance. These predictions are used to recommend an optimal configuration to users before launching their workloads in the cluster, avoiding possible failures, performance degradation and wastage of resources. We evaluate our method with a benchmark of 15 Spark applications on the Grid5000 testbed. We observe up to a 51% gain on performance when using the recommended parallelism settings. The model is also interpretable and can give insights to the user into how different metrics and parameters affect the performance.

show abstract

Recurring job optimization in scope

Cited by 22 publications

References 3 publications

Consolidation of queries with user-defined functions

Consolidation of queries with user-defined functions

Blink and it's done

Using machine learning to optimize parallelism in big data applications

Contact Info

Product

Resources

About