Connections Between Power Series Methods and Automatic Differentiation

Carothers, David C.; Lucas, Stephen K.; Parker, Gary; Rudmin, Joseph D.; Thelwell, Roger J.; Tongen, Anthony; Warne, Paul G.

doi:10.1007/978-3-642-30023-3_16

Cited by 6 publications

(7 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the context of machine learning, automatic differentiation can be seen as an an instance of generating functions [Carothers et al, 2012].…”

Section: George Pólya Mathematics and Plausible Reasoningmentioning

confidence: 99%

“…As such the residual stream serves a similar purpose to the evaluation trace in automatic differentiation, namely establishing recurrences. This is unsurprising considering the fact that automatic differentiation is a recurrence relationship based on formal power series [Carothers et al, 2012], in particular the Taylor series [Hoffmann, 2014].…”

Section: Residual Streammentioning

confidence: 99%

See 1 more Smart Citation

Coinductive guide to inductive transformer heads

Nemecek¹

2023

Preprint

View full text Add to dashboard Cite

We argue that all building blocks of transformer models can be expressed with a single concept: combinatorial Hopf algebra.Transformer learning emerges as a result of the subtle interplay between the algebraic and coalgebraic operations of the combinatorial Hopf algebra. Viewed through this lens, the transformer model becomes a linear time-invariant system where the attention mechanism computes a generalized convolution transform and the residual stream serves as a unit impulse.Attention-only transformers then learn by enforcing an invariant between these two paths. We call this invariant Hopf coherence. Due to this, with a degree of poetic license, one could call combinatorial Hopf algebras "tensors with a built-in loss function gradient". This loss function gradient occurs within the single layers and no backward pass is needed. This is in stark contrast to automatic differentiation which happens across the whole graph and needs a explicit backward pass. This property is the result of the fact that combinatorial Hopf algebras have the surprising property of calculating eigenvalues by repeated squaring.

show abstract

“…In the context of machine learning, automatic differentiation can be seen as an an instance of generating functions [Carothers et al, 2012].…”

Section: George Pólya Mathematics and Plausible Reasoningmentioning

confidence: 99%

Section: Residual Streammentioning

confidence: 99%

Coinductive guide to inductive transformer heads

Nemecek¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Parker and Sochacki [1] used this fact to generate solutions to nonlinear ODEs. In fact, the second method also works for nonlinear ODEs which was shown by Parker and Sochacki and exploited to give convergence and error results by Carothers et al [10] [11] [12]. The important point is that we now have two methods for developing algorithms and proving theorems for solutions to linear and nonlinear ODEs.…”

Section: Introduction Of the Psm Approach To Estimating Differential mentioning

confidence: 99%

Introducing the Power Series Method to Numerically Approximate Contingent Claim Partial Differential Equations

Buetow¹

2019

JMF

View full text Add to dashboard Cite

We introduce a previously unused numerical framework for estimating the Black-Scholes partial differential equation. The approach, known as the Power Series Method (PSM), offers several advantages over traditional finite difference methods. Our objective is to highlight the advantages of the PSM over traditionally used numerical approximation approaches. To meet this we deploy a numerical approximation scheme to illustrate the PSM. The PSM is more stable than explicit methods and thus computationally more efficient. It is as accurate as hybrid approaches like Crank Nicolson and faster to compute. It is more accurate over a far wider spectrum of time steps. Finally, and importantly, it can be expressed analytically thus offering the capability of performing comparative statics in a far more stable and accurate environment. For a more complex application this last advantage may have wide implications in producing hedge ratios for synthetic replication purposes.

show abstract

“…Aside from machine round-off error, PSM can apriori guarantee that, at any given step, the error of the approximation will remain less than a designated desired error tolerance [40]. For a brief overview of PSM, see Section 2 and [9,10].…”

mentioning

confidence: 99%

“…See [31,30,1,2,4,5,6,11,13,17,20,21,23,26,27,32,33,35,37,39] for a sample of recent studies that report the advantages (and some disadvantages) of PSM. PSM and work from the Automatic Differentiation (AD) community have developed nearly simultaneously [9,16,25], and there has been a number of benefits in collaborations between these two communities, including strategies in this work. Both [12] and [7] discuss the early AD foundation of interval analysis and generating the coefficients of a Taylor polynomial via successive derivatives and recurrence relations, which have clear connections to PSM.…”

mentioning

confidence: 99%

An Adaptive, Highly Accurate and Efficient, Parker-Sochacki Algorithm for Numerical Solutions to Initial Value Ordinary Differential Equation Systems

Guenther¹

2019

SIURO

View full text Add to dashboard Cite

The Parker-Sochacki Method (PSM) allows the numerical approximation of solutions to a polynomial initial value ordinary differential equation or system (IVODE) using an algebraic power series method. PSM is equivalent to a modified Picard iteration and provides an efficient, recursive computation of the coefficients of the Taylor polynomial at each step. To date, PSM has largely concentrated on fixed step methods. We develop and test an adaptive stepping scheme that, for many IVODEs, enhances the accuracy and efficiency of PSM. PSM Adaptive (PSMA) is compared to its fixed step counterpart and to standard Runge-Kutta (RK) foundation algorithms using three example IVODEs. In comparison, PSMA is shown to be competitive, often outperforming these methods in terms of accuracy, number of steps, and execution time. A library of functions is also presented that allows access to PSM techniques for many non-polynomial IVODEs without having to first rewrite these in the necessary polynomial form, making PSM a more practical tool.

show abstract

Connections Between Power Series Methods and Automatic Differentiation

Cited by 6 publications

References 5 publications

Coinductive guide to inductive transformer heads

Coinductive guide to inductive transformer heads

Introducing the Power Series Method to Numerically Approximate Contingent Claim Partial Differential Equations

An Adaptive, Highly Accurate and Efficient, Parker-Sochacki Algorithm for Numerical Solutions to Initial Value Ordinary Differential Equation Systems

Contact Info

Product

Resources

About