Cache-oblivious wavefront: improving parallelism of recursive dynamic programming algorithms without losing cache-efficiency

Abstract

State-of-the-art cache-oblivious parallel algorithms for dynamic programming (DP) problems usually guarantee asymptotically optimal cache performance without any tuning of cache parameters, but they often fail to exploit the theoretically best parallelism at the same time. While these algorithms achieve cache-optimality through the use of a recursive divide-and-conquer (DAC) strategy, scheduling tasks at the granularity of task dependency introduces artificial dependencies in addition to those arising from the defining recurrence equations. We removed the artificial dependency by scheduling tasks ready for execution as soon as all its real dependency constraints are satisfied, while preserving the cache-optimality by inheriting the DAC strategy. We applied our approach to a set of widely known dynamic programming problems, such as Floyd-Warshall's All-Pairs Shortest Paths, Stencil, and LCS. Theoretical analyses show that our techniques improve the span of 2-way DAC-based Floyd Warshall's algorithm on an n node graph from Θ n log 2 n to Θ (n), stencil computations on a d-dimensional hypercubic grid of width w for h time steps from Θ (d 2 h)w log(d+2)−1 to Θ (h), and LCS on two sequences of length n each from Θ n log 2 3 to Θ (n). In each case, the total work and cache complexity remain asymptotically optimal. Experimental measurements exhibit a 3 -5 times improvement in absolute running time, 10 -20 times improvement in burdened span by Cilkview, and approximately the same L1/L2 cache misses by PAPI.

Improved Parallel Cache-Oblivious Algorithms for Dynamic Programming [Extend Abstract]

“…For the problems that we consider in this paper, the parallel DP algorithms were already discussed by a rich literature in the eighties and nighties (e.g., [49,51,42,58,57,72]). Later work not only considers parallelism, but also optimizes symmetric cache complexity (e.g., [46,34,36,31,20,60,77,74,75,41,73,32]). The algorithms in linear algebra that share the similar computation structures (but with different orders in the computation) are also discussed (e.g., [36,41,83,78,25,40,11,65]).…”

“…For other problems (GAP, RNA, protein accordion folding, knapsack), the bounds in the symmetric setting are also improved. Some previous work [75,41] achieves the linear span in several problems. We note that they assume a much stronger model to guarantee the sequential and parallel execution order, so their algorithms need specially designed schedulers [41,30].…”

“…Polylogarithmic span can be achieved in computing the 2-knapsack recurrence, and linear span in LWS recurrence and protein accordion folding. The linear span for LWS can be achieved by previous work [75,41], but they are not race-free and in the nested-parallel model. Meanwhile, our algorithms are arguably simpler.…”

“…There exist work-optimal and sublinear depth algorithm for APSP [76], but we are unaware of how to make it I/O-efficient. Compared to previous linear-span algorithms [75,41], our new algorithms are race-free and in the nested-parallel model.…”

Experimental Analysis of Space-Bounded Schedulers

“…This overhead may be reduced to some extent by falsely reporting a greater size for Cholesky tasks at lower levels of recursion to force the SB scheduler to be more aggressive at load balancing. Further, Cholesky factorization could also achieve better performance through a relaxation of false dependencies introduced by expressing the algorithm in the fork-join paradigm using techniques recently introduced by Tang et al [2015]. This would reduce the depth of the algorithm to O(n/L) and remove all serial points in the DAG except the start and the end.…”