Generating performance distributions via probabilistic symbolic execution

Chen, Bihuan; Liu, Yang; Le, Wei

doi:10.1145/2884781.2884794

Cited by 50 publications

(30 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PerfPlotter [52] provides a performance analysis framework and a symbolic execution approach for generating performance distributions. It requires the source code of the program under test along with the usage profiles of the program and uses probabilistic symbolic execution to traverse different high-probability and low-probability execution paths in the program and to generate performance distributions.…”

Section: Related Workmentioning

confidence: 99%

Using Deep Reinforcement Learning for Exploratory Performance Testing of Software Systems With Multi-Dimensional Input Spaces

et al. 2020

View full text Add to dashboard Cite

During exploratory performance testing, software testers evaluate the performance of a software system with different input combinations in order to identify combinations that cause performance problems in the system under test. Performance problems such as low throughput, high response times, hangs, or crashes in software applications have an adverse effect on the customer's satisfaction. Since many of today's largescale, complex software systems (e.g., eCommerce applications, databases, web servers) exhibit very large multi-dimensional input spaces with many input parameters and large ranges, it has become costly and inefficient to explore all possible combinations of inputs in order to detect performance problems. In order to address this issue, we introduce a method for identifying input combinations that trigger performance problems in the software system under test. Our method, under the name of iPerfXRL, employs deep reinforcement learning in order to explore a given large multi-dimensional input space efficiently. The main benefit of the approach is that, during the exploration process, it learns and recognizes the problematic regions of the input space that have a higher chance of triggering performance problems. It concentrates the search in those problematic regions to find as many input combinations as possible that can trigger performance problems while executing a limited number of input combinations against the system. In addition, our approach does not require prior domain knowledge or access to the source code of the system. Therefore, it can be applied to any software system where we can interactively execute different input combinations while monitoring their performance impact on the system. We implement iPerfXRL on top of the Soft Actor-Critic algorithm. We evaluate empirically the efficiency and effectiveness of our approach against alternative state-of-the-art approaches. Our results show that iPerfXRL accurately identifies the problematic regions of the input space and finds up to 9 times more input combinations that trigger performance problems on the system under test than the alternative approaches. INDEX TERMS Exploratory performance testing, Deep reinforcement learning, Test data generation I. INTRODUCTION One of the most critical and challenging tasks for developers is to identify and fix performance problems of software systems [1]. Performance problems such as low throughput, high response times, hangs, or crashes in software applications have an adverse effect on the customer's satisfaction. According to [2], there are higher chances of a software system crashing due to performance problems rather than functional failures. Recent reports [1] show that certain input combinations can trigger more than half of the performance bottlenecks identified in non-trivial software systems. The reason is that certain input combinations can invoke inefficient code sequences or resource-intensive operations, which result in overall system performance degradation commonly referred to as performan...

show abstract

Section: Related Workmentioning

confidence: 99%

Using Deep Reinforcement Learning for Exploratory Performance Testing of Software Systems With Multi-Dimensional Input Spaces

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Thus, WISE does not scale to large complex programs. PerfPlotter [20] addresses this concern by probabilistically selecting paths to explore, using heuristics to find best-case and worst-case execution paths. Zhang et al [65] automatically generate load tests using mixed symbolic execution and iterativedeepening beam search.…”

Section: Related Workmentioning

confidence: 99%

PerfFuzz: automatically generating pathological inputs

Lemieux

Padhye

Sen

et al. 2018

Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis

131

View full text Add to dashboard Cite

Performance problems in software can arise unexpectedly when programs are provided with inputs that exhibit worst-case behavior. A large body of work has focused on diagnosing such problems via statistical profiling techniques. But how does one find these inputs in the first place? We present PerfFuzz, a method to automatically generate inputs that exercise pathological behavior across program locations, without any domain knowledge. Perf-Fuzz generates inputs via feedback-directed mutational fuzzing. Unlike previous approaches that attempt to maximize only a scalar characteristic such as the total execution path length, PerfFuzz uses multi-dimensional feedback and independently maximizes execution counts for all program locations. This enables PerfFuzz to (1) find a variety of inputs that exercise distinct hot spots in a program and (2) generate inputs with higher total execution path length than previous approaches by escaping local maxima. PerfFuzz is also effective at generating inputs that demonstrate algorithmic complexity vulnerabilities. We implement PerfFuzz on top of AFL, a popular coverage-guided fuzzing tool, and evaluate PerfFuzz on four real-world C programs typically used in the fuzzing literature. We find that PerfFuzz outperforms prior work by generating inputs that exercise the most-hit program branch 5× to 69× times more, and result in 1.9× to 24.7× longer total execution paths. CCS CONCEPTS • Software and its engineering → Software testing and debugging; Software performance;

show abstract

“…Probabilistic symbolic execution is used by Chen et al [13] to infer the performance distribution of a program according to given usage profiles. The technique aims to obtain a diverse set of program behaviours by guiding the execution along high-and low-probability program branches; it further uses loop unrolling to try to force the longest execution through the loops.…”

Section: Related Workmentioning

confidence: 99%

Complexity vulnerability analysis using symbolic execution

Luckow

Kersten

Păsăreanu

2020

Software Testing Verif & Rel

View full text Add to dashboard Cite

We describe techniques based on symbolic execution for finding software vulnerabilities that are due to algorithmic complexity. Such vulnerabilities allow an attacker to mount denial-of-service attacks to deny service to benign users or to otherwise disable a software system. The techniques use an efficient guided symbolic execution of a program to compute bounds on the worstcase complexity (for increasing input sizes) and to generate test values that trigger the worst-case behaviours. The resulting bounds are fitted to a function to obtain a prediction of the worst-case program behaviour at any input size. Scalability is achieved by using path policies that guide the symbolic execution towards worst-case paths. The policies are learned from the worst-case results obtained with exhaustive exploration at small input sizes and are applied to guide exploration at larger input sizes, where unguided exhaustive exploration is not possible. To achieve precision in the analysis, the path policies take into account the history of choices made along the path when deciding which branch to execute next. Furthermore, the computation is contextpreserving, meaning that the decision for each branch depends on the history computed with respect to the enclosing method. We further report preliminary results on a complementary technique that uses machine learning for building the path policies that guide the search. The techniques are implemented in open-source projects that build on the Symbolic Pathfinder tool for analysing Java programs. Experimental evaluation shows that the techniques can find vulnerabilities in complex Java programs and can outperform previous symbolic approaches.

show abstract

Generating performance distributions via probabilistic symbolic execution

Cited by 50 publications

References 54 publications

Using Deep Reinforcement Learning for Exploratory Performance Testing of Software Systems With Multi-Dimensional Input Spaces

Using Deep Reinforcement Learning for Exploratory Performance Testing of Software Systems With Multi-Dimensional Input Spaces

PerfFuzz: automatically generating pathological inputs

Complexity vulnerability analysis using symbolic execution

Contact Info

Product

Resources

About