Functional programming for dynamic and large data with self-adjusting computation

Chen, Yan; Acar, Umut A.; Tangwongsan, Kanat

doi:10.1145/2628136.2628150

Cited by 6 publications

(3 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the previous section we explored options for partial recomputation of a previously executed process. Here we look at a complementary option, namely re-computing P using only the differences diff D (d t , d t ) between two versions of (one or more) reference dataset, D. Some of these ideas are grounded in prior research on the incremental computation and differential computation domains [22,10].…”

Section: Differential Executionmentioning

confidence: 99%

Selective and Recurring Re-computation of Big Data Analytics Tasks: Insights from a Genomics Case Study

Cała

Missier

2018

Big Data Research

View full text Add to dashboard Cite

In Data Science, knowledge generated by a resource-intensive analytics process is a valuable asset. Such value, however, tends to decay over time as a consequence of the evolution of any of the elements the process depends on: external data sources, libraries, and system dependencies. It is therefore important to be able to (i) detect changes that may partially or completely invalidate prior outcomes, (ii) determine the impact that those changes will have on those prior outcomes, ideally without having to perform expensive re-computations, and (iii) optimise the process re-execution needed to selectively refresh affected outcomes. This paper presents an extensive experimental study on how the selective re-computation problem manifests itself in a relevant analytics task for Genomics, namely variant calling and clinical interpretation, and how the problem can be addressed using a combination of approaches. Starting from this experience, we then offer a blueprint for a generic re-computation meta-process that makes use of process history metadata to make informed decisions about selective recomputations in reaction to a variety of changes in the data.

show abstract

Section: Differential Executionmentioning

confidence: 99%

Selective and Recurring Re-computation of Big Data Analytics Tasks: Insights from a Genomics Case Study

Cała

Missier

2018

Big Data Research

View full text Add to dashboard Cite

show abstract

“…Bhatotia et al (2011) extend memoization to distributed, cloudbased settings (MapReduce-style computations in particular). Chen et al (2014) reduce the (often large) time and space overhead, which is pervasive in both SAC and in ADAPTON. In particular, they propose coarsening the granu-larity of dependence tracking, and report massive reductions (orders of magnitude) in space as a result.…”

Section: Related Workmentioning

confidence: 99%

Incremental computation with names

Hammer

Dunfield

Headley

et al. 2015

Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applicatio

View full text Add to dashboard Cite

Over the past thirty years, there has been significant progress in developing general-purpose, language-based approaches to incremental computation, which aims to efficiently update the result of a computation when an input is changed. A key design challenge in such approaches is how to provide efficient incremental support for a broad range of programs. In this paper, we argue that first-class names are a critical linguistic feature for efficient incremental computation. Names identify computations to be reused across differing runs of a program, and making them first class gives programmers a high level of control over reuse. We demonstrate the benefits of names by presenting NOMI-NAL ADAPTON, an ML-like language for incremental computation with names. We describe how to use NOMINAL ADAPTON to efficiently incrementalize several standard programming patterns-including maps, folds, and unfoldsand show how to build efficient, incremental probabilistic trees and tries. Since NOMINAL ADAPTON's implementation is subtle, we formalize it as a core calculus and prove it is from-scratch consistent, meaning it always produces the same answer as simply re-running the computation. Finally, we demonstrate that NOMINAL ADAPTON can provide large speedups over both from-scratch computation and ADAPTON, a previous state-of-the-art incremental computation system.

show abstract

“…Dynamic dependencies One of the recent developments in incremental computing is that of so-called self-adjusting computation (Chen et al 2014). In self-adjusting computation all data is labelled with either static or changeable, and special constructs are added to the language in order to handle changeable data in such a way that changes can be efficiently propagated.…”

Section: Related Workmentioning

confidence: 99%

Incremental Evaluation of Higher Order Attributes

Bransen

Dijkstra

Swierstra

2015

Proceedings of the 2015 Workshop on Partial Evaluation and Program Manipulation

View full text Add to dashboard Cite

Compilers, amongst other programs, often work with data that (slowly) changes over time. When the changes between subsequent runs of the compiler are small, one would hope the compiler to incrementally update its results, resulting in much lower running times. However, the manual construction of an incremental compiler is very hard and error prone and therefore usually not an option.Attribute grammars provide an attractive way of constructing compilers, as they are compositional in nature and allow for aspect oriented programming. In this work we extend previous work on the automatic generation of incremental attribute grammar evaluators, with the purpose of (semi-)automatically generating an incremental compiler from the regular attribute grammar definition, by adding support for incremental evaluation of higher order attributes, a well known extension to the classical attribute grammars that is used in many ways in compiler construction, for example to model different compiler phases.

show abstract

Functional programming for dynamic and large data with self-adjusting computation

Cited by 6 publications

References 59 publications

Selective and Recurring Re-computation of Big Data Analytics Tasks: Insights from a Genomics Case Study

Selective and Recurring Re-computation of Big Data Analytics Tasks: Insights from a Genomics Case Study

Incremental computation with names

Incremental Evaluation of Higher Order Attributes

Contact Info

Product

Resources

About