A first-order one-pass CPS transformation

Danvy, Olivier; Nielsen, Lasse R.

doi:10.1016/s0304-3975(02)00733-8

Cited by 25 publications

(21 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Delimited Continuations: A Simpler Essence. Lantern relies on delimited continuations Filinski 1990, 1992;Danvy and Nielsen 2003], as implemented in Scala [Rompf et al 2009]. In parallel to our work, which first appeared as tech report on arXiv [Wang et al 2018b], Elliott [2018] proposed a generalized view of AD based on the paradigm of "compiling to categories" [Elliott 2017].…”

Section: Related Workmentioning

confidence: 96%

Demystifying differentiable programming: shift/reset the penultimate backpropagator

Wang

Zheng

Decker

et al. 2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

Deep learning has seen tremendous success over the past decade in computer vision, machine translation, and gameplay. This success rests crucially on gradient-descent optimization and the ability to "learn" parameters of a neural network by backpropagating observed errors. However, neural network architectures are growing increasingly sophisticated and diverse, which motivates an emerging quest for even more general forms of differentiable programming, where arbitrary parameterized computations can be trained by gradient descent. In this paper, we take a fresh look at automatic differentiation (AD) techniques, and especially aim to demystify the reverse-mode form of AD that generalizes backpropagation in neural networks.We uncover a tight connection between reverse-mode AD and delimited continuations, which permits implementing reverse-mode AD purely via operator overloading and without managing any auxiliary data structures. We further show how this formulation of AD can be fruitfully combined with multi-stage programming (staging), leading to an efficient implementation that combines the performance benefits of deep learning frameworks based on explicit reified computation graphs (e.g., TensorFlow) with the expressiveness of pure library approaches (e.g., PyTorch).function [Rumelhart et al. 1986]. Beyond this commonality, however, deep learning architectures vary widely. In fact, many of the practical successes are fueled by increasingly sophisticated and diverse network architectures that in many cases depart from the traditional organization into layers of artificial neurons. For this reason, prominent deep learning researchers have called for a paradigm shift from deep learning towards differentiable programming [LeCun 2018; Olah 2015] -essentially, functional programming with first-class gradients -based on the expectation that further advances in artificial intelligence will be enabled by the ability to "train" arbitrary parameterized computations by gradient descent.Programming language designers and compiler writers, key players in this vision, are faced with the challenge of adding efficient and expressive program differentiation capabilities. Forms of automatic gradient computation that generalize the classic backpropagation algorithm are provided by all contemporary deep learning frameworks, including TensorFlow and PyTorch. These implementations, however, are ad-hoc, and each framework comes with its own set of trade-offs and restrictions. In the academic world, automatic differentiation (AD) [Speelpenning 1980;Wengert 1964] is the subject of study of an entire community. Unfortunately, results disseminate only slowly between communities, and while the forward-mode flavor of AD is easy to grasp, descriptions of the reverse-mode flavor that generalizes backpropagation often appear mysterious to PL researchers. A notable exception is the seminal work of Pearlmutter and Siskind [2008], which cast AD in a functional programming framework and laid the groundwork for first-class, unrestricted, gradient ope...

show abstract

Section: Related Workmentioning

confidence: 96%

Demystifying differentiable programming: shift/reset the penultimate backpropagator

Wang

Zheng

Decker

et al. 2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

show abstract

“…The complete proof appears in the full version of this article [13]. It is more direct than Plotkin's [30] since we do not need a colon translation.…”

Section: Theorem 1 (Simulation) a Term E Reduces To A Value T If Andmentioning

confidence: 99%

A First-Order One-Pass CPS Transformation

Danvy

Nielsen

2002

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

We present a new transformation of call-by-value lambdaterms into continuation-passing style (CPS). This transformation operates in one pass and is both compositional and first-order. Because it operates in one pass, it directly yields compact CPS programs that are comparable to what one would write by hand. Because it is compositional, it allows proofs by structural induction. Because it is first-order, reasoning about it does not require the use of a logical relation. This new CPS transformation connects two separate lines of research. It has already been used to state a new and simpler correctness proof of a direct-style transformation, and to develop a new and simpler CPS transformation of control-flow information. Basic Research in Computer Science (www.brics.dk), funded by the Danish National Research Foundation.

show abstract

“…A noncompositional transformation requires well-founded induction rather than ordinary structural induction [40]. Fortunately, Danvy and Nielsen have recently discovered a one-pass CPS transformation that is both first-order and compositional [14,30].…”

Section: Cps Transformation Of Termsmentioning

confidence: 99%

“…In this article, we build on Danvy and Nielsen's new one-pass CPS transformation [14,30] and we present a new and simpler CPS transformation of control-flow information that does not construct any intermediate result and thus is more efficient to use. It is also simpler to prove correct.…”

Section: Cps Transformation Of Flow Informationmentioning

confidence: 99%

See 1 more Smart Citation

A Simple CPS Transformation of Control-Flow Information

Damian¹

2002

Logic Journal of IGPL

View full text Add to dashboard Cite

We build on Danvy and Nielsen's first-order program transformation into continuation-passing style (CPS) to design a new CPS transformation of flow information that is simpler and more efficient than what has been presented in previous work. The key to simplicity and efficiency is that our CPS transformation constructs the flow information in one go, instead of first computing an intermediate result and then exploiting it to construct the flow information.More precisely, we show how to compute control-flow information for CPStransformed programs from control-flow information for direct-style programs and vice-versa. As a corollary, we confirm that CPS transformation has no effect on the control-flow information obtained by constraint-based control-flow analysis. The transformation has immediate applications in assessing the effect of the CPS transformation over other analyses such as, for instance, binding-time analysis.

show abstract

A first-order one-pass CPS transformation

Cited by 25 publications

References 35 publications

Demystifying differentiable programming: shift/reset the penultimate backpropagator

Demystifying differentiable programming: shift/reset the penultimate backpropagator

A First-Order One-Pass CPS Transformation

A Simple CPS Transformation of Control-Flow Information

Contact Info

Product

Resources

About