Main memory evaluation of recursive queries on multicore machines

Yang, Mohan; Zaniolo, Carlo

doi:10.1109/bigdata.2014.7004240

Cited by 8 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our ongoing work seeks to further optimize the code generation which reduces the performance gap between DeALS and the hand written optimal programs, including (i) employing a vectorized processing model [4] and the techniques presented in [12] to improve the performance on non-recursive queries; (ii) implementing the SSC12 algorithm [56] for transitive closure-like recursive queries; and (iii) providing a worst-case optimal guarantee for joins used by both nonrecursive and recursive queries with the leapfrog triejoin algorithm [51]. Another improvement planned for the future is to study techniques that can be integrated into DeALS to improve its performance when skew is present.…”

Section: Resultsmentioning

confidence: 99%

Scaling up the performance of more powerful Datalog systems on multicore machines

2016

Self Cite

View full text Add to dashboard Cite

Extending RDBMS technology to achieve performance and scalability for queries that are much more powerful than those of SQL-2 has been the goal of deductive database research for more than thirty years. The DeALS system has made major progress toward this goal, by (1) Datalog extensions that support the more powerful recursive queries needed in advanced applications, and (2) superior performance for both traditional recursive queries and those made possible by the new extensions, while (3) delivering competitive performance with commercial RDBMSs on non-recursive queries. In this paper, we focus on the techniques used to support the in-memory evaluation of Datalog programs on multicore machines. In DeALS, a Datalog program is represented as an AND/OR tree, and multiple copies of the same AND/OR tree are used to access the tables in the database concurrently during the parallel evaluation. We describe compilation techniques that (1) recognize when the given program is lock-free, (2) transform a locking program into a lock-free program, and (3) find an efficient parallel plan that correctly evaluates the program while minimizing the use of locks and other overhead required for parallel evaluation. Extensive experiments demonstrate the effectiveness of the proposed techniques.

show abstract

Section: Resultsmentioning

confidence: 99%

Scaling up the performance of more powerful Datalog systems on multicore machines

2016

Self Cite

View full text Add to dashboard Cite

show abstract

“…[21] showed how XY-stratified Datalog can support computational models for large-scale machine learning, although no full Datalog language implementation on a large-scale system was provided. Recent works on recursive query evaluation showed efficient versions of transitive closure for multi-core [60] and distributed [14] settings, however these works did not address how to convert arbitrary programs to these desirable evaluation forms.…”

Section: Relatedworkmentioning

confidence: 99%

Big Data Analytics with Datalog Queries on Spark

Shkapsky

Yang

Interlandi

et al. 2016

Proceedings of the 2016 International Conference on Management of Data

Self Cite

View full text Add to dashboard Cite

There is great interest in exploiting the opportunity provided by cloud computing platforms for large-scale analytics. Among these platforms, Apache Spark is growing in popularity for machine learning and graph analytics. Developing efficient complex analytics in Spark requires deep understanding of both the algorithm at hand and the Spark API or subsystem APIs (e.g., Spark SQL, GraphX). Our system addresses the problem by providing concise declarative specification of complex queries amenable to efficient evaluation. Towards this goal, we propose compilation and optimization techniques that tackle the important problem of efficiently supporting recursion in Spark. We perform an experimental comparison with other state-of-the-art large-scale Datalog systems and verify the efficacy of our techniques and effectiveness of Spark in supporting Datalog-based analytics.

show abstract

“…A growing body of research on scalable data analytics has brought a renaissance of interest in Datalog because of its ability to specify declaratively advanced dataintensive applications that execute efficiently over different systems and architectures, including massively parallel ones Shkapsky et al 2013;Yang and Zaniolo 2014;Aref et al 2015;Wang et al 2015;Yang et al 2015;Shkapsky et al 2016;Yang et al 2017). The trends and developments that have led to this renaissance can be better appreciated if we contrast them with those that motivated the early research on Datalog back in the 80s.…”

Section: Introductionmentioning

confidence: 99%

Scaling-up reasoning and advanced analytics on BigData

Condie¹,

Das²,

Interlandi³

et al. 2018

Theory and Practice of Logic Programming

Self Cite

View full text Add to dashboard Cite

BigDatalog is an extension of Datalog that achieves performance and scalability on both Apache Spark and multicore systems to the point that its graph analytics outperform those written in GraphX. Looking back, we see how this realizes the ambitious goal pursued by deductive database researchers beginning 40 years ago: this is the goal of combining the rigor and power of logic in expressing queries and reasoning with the performance and scalability by which relational databases managed BigData. This goal led to Datalog which is based on Horn Clauses like Prolog but employs implementation techniques, such as semi-naïve fixpoint and magic sets, that extend the bottom-up computation model of relational systems, and thus obtain the performance and scalability that relational systems had achieved, as far back as the 80s, using data-parallelization on shared-nothing architectures. But this goal proved difficult to achieve because of major issues at (i) the language level and (ii) at the system level. The paper describes how (i) was addressed by simple rules under which the fixpoint semantics extends to programs using count, sum and extrema in recursion, and (ii) was tamed by parallel compilation techniques that achieve scalability on multicore systems and Apache Spark. This paper is under consideration for acceptance in Theory and Practice of Logic Programming.

show abstract

Main memory evaluation of recursive queries on multicore machines

Cited by 8 publications

References 21 publications

Scaling up the performance of more powerful Datalog systems on multicore machines

Scaling up the performance of more powerful Datalog systems on multicore machines

Big Data Analytics with Datalog Queries on Spark

Scaling-up reasoning and advanced analytics on BigData

Contact Info

Product

Resources

About