Consolidation of queries with user-defined functions

Abstract

Motivated by streaming and data analytics scenarios where many queries operate on the same data and perform similar computations, we propose program consolidation for merging multiple user-defined functions (UDFs) that operate on the same input. Program consolidation exploits common computations between UDFs to generate an equivalent optimized function whose execution cost is often much smaller (and never greater) than the sum of the costs of executing each function individually. We present a sound consolidation calculus and an effective algorithm for consolidating multiple UDFs. Our approach is purely static and uses symbolic SMTbased techniques to identify shared or redundant computations. We have implemented the proposed technique on top of the Naiad data processing system. Our experiments show that our algorithm dramatically improves overall job completion time when executing user-defined filters that operate on the same data and perform similar computations.

Cartesian hoare logic for verifying k-safety properties

“…Product programs. Our work is most closely related to a line of work on product programs, which have been used in the context of translation validation [18], proving noninterference [3,5] and program optimizations [21]. Given two programs A, B (with disjoint variables), these approaches explicitly construct a new program, called the product of A and B, that is semantically equivalent to A; B but is somehow easier to verify.…”

“…. Rn, Ψ); let χi := REMOVEITH(χ, i) in 11: if(!K-VERIFY(Φ , χi, Ψ)) return false; , B, )] * ; R ) then Flatten 15: return K-VERIFY(Φ, χ [(c, B, )] * ; R ; R, Ψ); 16: else 17: let I := LOOPINV([L] * ) in Loop 18: foreach (i ∈ [1,n]) 19: let Φ := post i (I) ∧ ¬ci; 20:if(!K-VERIFY(Φ , S i ; R χ ))21:then return false;22:…”

Automated Hypersafety Verification

“…1. Product programs generalize the self-composition approach and have been used in verifying translation validation [20], non-interference [16,23], and program optimization [25]. A product of two programs P 1 and P 2 is semantically equivalent to P 1 • P 2 (sequential composition), but is made easier to verify by allowing parts of each program to be interleaved.…”

Relational program synthesis

“…In contrast to these approaches that provide a dedicated program logic for reasoning about relational properties, an alternative approach is to build a so-called product program that captures the simultaneous execution of multiple programs or different runs of the same program [Barthe et al 2011[Barthe et al , 2013[Barthe et al , 2004. In the simplest case, this approach sequentially composes different programs (or copies of the same program) [ Barthe et al 2004], but more sophisticated product construction methods perform various transformations such as loop fusion and unrolling to make invariant generation easier [Barthe et al 2011[Barthe et al , 2013Sousa et al 2014;Zaks and Pnueli 2008]. A common theme underlying all these product construction techniques is to reduce the relational verification problem to a standard safety checking problem.…”