Searching for More Efficient Dynamic Programs

Abstract

Computational models of human language often involve combinatorial problems. For instance, a probabilistic parser may marginalize over exponentially many trees to make predictions. Algorithms for such problems often employ dynamic programming and are not always unique. Finding one with optimal asymptotic runtime can be unintuitive, timeconsuming, and error-prone. Our work aims to automate this laborious process. Given an initial correct declarative program, we search for a sequence of semantics-preserving transformations to improve its running time as much as possible. To this end, we describe a set of program transformations, a simple metric for assessing the efficiency of a transformed program, and a heuristic search procedure to improve this metric. We show that in practice, automated search-like the mental search performed by human programmers-can find substantial improvements to the initial program. Empirically, we show that many speed-ups described in the NLP literature could have been discovered automatically by our system.* word(Y,I,K). z += β(root,0,N) * len(N). start β(X,I,K) += tmp(I,J,X,Z) * β(Z,J,K). β(X,I,K) += γ(X,Y) * word(Y,I,K). z += β(root,0,N) * len(N). tmp(I,J,X,Z) += β(Y,I,J) * γ(X,Y,Z).fold(1, [1, 2]) elim( 4)* word(Y,I,K). z += β(root,0,N) * len(N). tmp(I,J,X,Z) += β(Y,I,J) * γ(X,Y,Z). 21α("a") max= 1.22 α(J) max= α(I) * w(I,J). 23z += α("z").The second rule is recursive. For each possible value of J on the left-hand side, it defines α(J) by maximizing over assignments to the other variables on the right-hand side (namely I). Maximization is specified by max=, whereas summation in Example 1 was specified by +=.

Dynamic Programming in Rank Space: Scaling Structured Inference with Low-Rank HMMs and PCFGs

“…In fact, this is a typical application of the unfold-refold transformation(Eisner and Blatz, 2007;Vieira et al, 2021).5 This does not take into account the one-time cost of computing H, I, J, K before inference.6 The Viterbi semiring is also known as the max-product semiring.Chiu et al (2021, Appd. C) andYang et al (2021b, Sec.…”