Measuring empirical computational complexity

Abstract

We propose a method for describing the asymptotic behavior of programs in practice by measuring their empirical computational complexity. Our method involves running a program on workloads spanning several orders of magnitude in size, measuring their performance, and fitting these observations to a model that predicts performance as a function of workload size. Comparing these models to the programmer's expectations or to theoretical asymptotic bounds can reveal performance bugs or confirm that a program's performance 2 scales as expected.We develop our methodology for constructing these models of empirical complexity as we describe and evaluate two techniques. Our first technique, BB-TrendProf, constructs models that predict how many times each basic block runs as a linear (y = a + bx) or a powerlaw (y = ax b ) function of user-specified features of the program's workloads. To present output succinctly and focus attention on scalability-critical code, BB-TrendProf groups and ranks program locations based on these models. We demonstrate the power of BB-TrendProf compared to existing tools by running it on several large programs and reporting cases where its models show (1) an implementation of a complex algorithm scaling as expected, (2) two complex algorithms beating their worst-case theoretical complexity bounds when run on realistic inputs, and (3) a performance bug.Our second technique, CF-TrendProf, models performance of loops and functions both per-function-invocation and per-workload. It improves upon the precision of BB-TrendProf's models by using control flow to generate candidates from a richer family of models and a novel model selection criteria to select among these candidates. We show that CF-TrendProf's improvements to model generation and selection allow it to correctly characterize or closely approximate the empirical scalability of several well-known algorithms and data structures and to diagnose several synthetic, but realistic, scalability problems without observing an egregiously expensive workload. We also show that CF-TrendProf deals with multiple workload features better than BB-TrendProf. We qualitatively compare the output of BB-TrendProf and CF-TrendProf and discuss their relative strengths and weaknesses. ii

Context-sensitive delta inference for identifying workload-dependent performance bottlenecks

“…For the least-squares regression used by our approach, a guideline for selecting the initial values is to avoid selecting a very small workload, such as 1 or 2 files, as the initial workload for a file manager. We empirically find that such small workloads produce noise in the inferred models, consistent with the finding by Goldsmith et al [19].…”

“…Existing approaches [16,19] that infer complexity models using profiles of multiple workloads are context-insensitive. To show effectiveness of context-sensitive analysis and answer RQ3, we compare the number of complexity transitions identified by using context-sensitive analysis and contextinsensitive analysis.…”

“…Goldsmith et al [19] propose an approach that fits performance measurements of clusters of basic blocks to workload sizes. Zaparanuks et al [38] propose an approach that infers an empirical cost function of an application automatically, and Coppa et al [16] propose an approach that measures the size of the input given to a generic code fragment.…”

“…To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ISSTA '13, July [15][16][17][18][19][20]2013, Lugano, Switzerland Copyright 13 ACM 978-1-4503-2159-4/13/07 ...$10.00.…”

The road not taken: Estimating path execution frequency statically

“…Current work in computational complexity estimation has been limited to run-time analyses [17]. At a high level, the task of estimating the complexity of the put procedure in Figure 1 requires identifying the paths that represent the common case and thus dominate the long-run runtime behavior.…”