Incremental computation with names

Hammer, Matthew A.; Dunfield, Joshua; Headley, Kyle; Labich, Nicholas; Foster, Jeffrey S.; Hicks, Michael; Horn, David Van

doi:10.1145/2814270.2814305

Cited by 27 publications

(14 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most incremental systems, e.g. [3,4,22] evaluate the program and use programmer-supplied information to build a dependency graph, whereas our system uses static code analysis; [23] implement a small adaptive interpreter that treats code as changing input data, suggesting an implementation technique for live programming systems. Our use of dependency graphs [32] is static and first-order and can be seen as a form of program slicing [59], although our binding process is more directly inspired by Roslyn [40], which uses it for efficient background type-checking.…”

Section: Related and Future Workmentioning

confidence: 99%

Foundations of a live data exploration environment

Petříček

2020

Programming

View full text Add to dashboard Cite

A growing amount of code is written to explore and analyze data, often by data analysts who do not have a traditional background in programming, for example by journalists. The way such data anlysts write code is different from the way software engineers do so. They use few abstractions, work interactively and rely heavily on external libraries. We aim to capture this way of working and build a programming environment that makes data exploration easier by providing instant live feedback.We combine theoretical and applied approach. We present the data exploration calculus, a formal language that captures the structure of code written by data analysts. We then implement a data exploration environment that evaluates code instantly during editing and shows previews of the results. We formally describe an algorithm for providing instant previews for the data exploration calculus that allows the user to modify code in an unrestricted way in a text editor. Supporting interactive editing is tricky as any edit can change the structure of code and fully recomputing the output would be too expensive. We prove that our algorithm is correct and that it reuses previous results when updating previews after a number of common code edit operations. We also illustrate the practicality of our approach with an empirical evaluation and a case study.As data analysis becomes an ever more important use of programming, research on programming languages and tools needs to consider new kinds of programming workflows appropriate for those domains and conceive new kinds of tools that can support them. The present paper is one step in this important direction. ACM CCS 2012Human-centered computing → Interactive systems and tools; Information systems → Data mining;Software and its engineering → Compilers;Foundations of a live data exploration environment development and execution. Data analysts write small snippets of code, run them to see results immediately and then revise them.Notebooks are used by users ranging from scientists who implement complex models of physical systems to journalists who perform simple data aggregations and create visualizations. Our focus is on the simplest use cases. Making programmatic data exploration more spreadsheet-like should encourage users to choose programming tools over spreadsheets, resulting in more reproducible and transparent data analyses.Consider the Financial Times analysis of plastic waste [7,25]. It joins datasets from Eurostat, UN Comtrade and more, aggregates the data and builds a visualization comparing waste flows in 2017 and 2018. Figure 1 shows an excerpt from one notebook of the data analysis. The code has a number of important properties:There is no abstraction. The analysis uses lambda functions as arguments to library calls, but it does not define any custom functions. Code is parameterized by having a global variable material set to "plastics" and keeping other possible values in a comment. This lets the analyst run and check results of intermediate steps.The code relies on external lib...

show abstract

Section: Related and Future Workmentioning

confidence: 99%

Foundations of a live data exploration environment

Petříček

2020

Programming

View full text Add to dashboard Cite

show abstract

“…As usual, incrementalization is not efficient on arbitrary programs, but only on programs designed so that input changes produce small changes to the computation trace; refinement type systems have been designed to assist in this task [8,12]. To identify matching inputs, Nominal Adapton [13] replaces input comparisons by pointer equality with first-class labels, enabling more reuse.…”

Section: Related Workmentioning

confidence: 99%

Incremental $$\lambda $$ -Calculus in Cache-Transfer Style

Giarrusso

Régis-Gianas

Schuster

2019

Programming Languages and Systems

View full text Add to dashboard Cite

Incremental computation requires propagating changes and reusing intermediate results of base computations. Derivatives, as produced by static differentiation [7], propagate changes but do not reuse intermediate results, leading to wasteful recomputation. As a solution, we introduce conversion to Cache-Transfer-Style, an additional program transformations producing purely incremental functional programs that create and maintain nested tuples of intermediate results. To prove CTS conversion correct, we extend the correctness proof of static differentiation from STLC to untyped λ-calculus via step-indexed logical relations, and prove sound the additional transformation via simulation theorems.To show ILC-based languages can improve performance relative to from-scratch recomputation, and that CTS conversion can extend its applicability, we perform an initial performance case study. We provide derivatives of primitives for operations on collections and incrementalize selected example programs using those primitives, confirming expected asymptotic speedups.

show abstract

“…Adapton [8,9] is a library for on-demand (lazy) incremental computation. Like the Pluto and our hybrid algorithm, Adapton supports a form of dynamic task dependencies: dynamic computation dependencies which form a computation graph.…”

Section: Related Workmentioning

confidence: 99%

Scalable incremental building with dynamic task dependencies

Konat

Erdweg

Visser

2018

Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering

View full text Add to dashboard Cite

Incremental build systems are essential for fast, reproducible software builds. Incremental build systems enable short feedback cycles when they capture dependencies precisely and selectively execute build tasks efficiently. A much overlooked feature of build systems is the expressiveness of the scripting language, which directly influences the maintainability of build scripts. In this paper, we present a new incremental build algorithm that allows build engineers to use a full-fledged programming language with explicit task invocation, value and file inspection facilities, and conditional and iterative language constructs. In contrast to prior work on incrementality for such programmable builds, our algorithm scales with the number of tasks affected by a change and is independent of the size of the software project being built. Specifically, our algorithm accepts a set of changed files, transitively detects and re-executes affected build tasks, but also accounts for new task dependencies discovered during building. We have evaluated the performance of our algorithm in a real-world case study and confirm its scalability. CCS CONCEPTS • Software and its engineering → Software configuration management and version control systems;

show abstract

Incremental computation with names

Cited by 27 publications

References 36 publications

Foundations of a live data exploration environment

Foundations of a live data exploration environment

Incremental $$\lambda $$ -Calculus in Cache-Transfer Style

Scalable incremental building with dynamic task dependencies

Contact Info

Product

Resources

About