PySE: Automatic Worst-Case Test Generation by Reinforcement Learning

Koo, Jinkyu; Saumya, Charitha; Kulkarni, Milind; Bagchi, Saurabh

doi:10.1109/icst.2019.00023

Cited by 23 publications

(18 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this case, Algorithm 2 can be used to generate positive tests for the new version using the old generator model. Otherwise, the discriminator model is trained with the newly executed tests and it will contest with the generator model until the FJD distance between the synthetic tests and the executed ones becomes low again (lines [13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28]. The contest procedure is the same as that of Algorithm 1.…”

Section: Acta Applied To Devopsmentioning

confidence: 99%

“…However, finding those values is mostly manual, intellectually intensive and laborious [30]. In recent years, researchers have proposed search-based profilers [21], [30], fuzzers [17], symbolic execution methods [6], [7], [26], and machine learning (ML) methods [14], [20], [25] to find those input values automatically and in a cost-effective way. However, most of these methods (e.g.…”

Section: Introductionmentioning

confidence: 99%

“…[7], [21], [26], [30]) are suitable for whitebox settings, where we have access to the source code of the SUT, which is not true in most cases [1]. Methods based on symbolic execution do not scale to systems with a large set of input data, since the number of paths to search grows exponentially with respect to the input size [14].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Automated Performance Testing Based on Active Deep Learning

Sedaghatbaf¹,

Moghadam²,

Saadatmand³

2021

Preprint

View full text Add to dashboard Cite

Generating tests that can reveal performance issues in large and complex software systems within a reasonable amount of time is a challenging task. On one hand, there are numerous combinations of input data values to explore. On the other hand, we have a limited test budget to execute tests. What makes this task even more difficult is the lack of access to source code and the internal details of these systems. In this paper, we present an automated test generation method called ACTA for black-box performance testing. ACTA is based on active learning, which means that it does not require a large set of historical test data to learn about the performance characteristics of the system under test. Instead, it dynamically chooses the tests to execute using uncertainty sampling. ACTA relies on a conditional variant of generative adversarial networks, and facilitates specifying performance requirements in terms of conditions and generating tests that address those conditions. We have evaluated ACTA on a benchmark web application, and the experimental results indicate that this method is comparable with random testing, and two other machine learning methods, i.e. PerfXRL and DN.

show abstract

Section: Acta Applied To Devopsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Automated Performance Testing Based on Active Deep Learning

Sedaghatbaf¹,

Moghadam²,

Saadatmand³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Machine learning has been also applied to the generation of performance test cases in some studies. For example, using symbolic execution in combination with an RL algorithm to find the worst-case execution path within a SUT (Koo et al 2019), using RL to find a sequence of input workload leading to performance degradation (Ahmad et al 2019), and feedback-driven learning to identify the performance bottlenecks through extracting rules from execution traces (Grechanik, Fu and Xie 2012). There are also some adaptive techniques slightly analogous to the concept of RL for generating performance test cases.…”

Section: Related Workmentioning

confidence: 99%

An autonomous performance testing framework using self-adaptive fuzzy reinforcement learning

et al. 2021

View full text Add to dashboard Cite

Test automation brings the potential to reduce costs and human effort, but several aspects of software testing remain challenging to automate. One such example is automated performance testing to find performance breaking points. Current approaches to tackle automated generation of performance test cases mainly involve using source code or system model analysis or use-case-based techniques. However, source code and system models might not always be available at testing time. On the other hand, if the optimal performance testing policy for the intended objective in a testing process instead could be learned by the testing system, then test automation without advanced performance models could be possible. Furthermore, the learned policy could later be reused for similar software systems under test, thus leading to higher test efficiency. We propose SaFReL, a self-adaptive fuzzy reinforcement learning-based performance testing framework. SaFReL learns the optimal policy to generate performance test cases through an initial learning phase, then reuses it during a transfer learning phase, while keeping the learning running and updating the policy in the long term. Through multiple experiments in a simulated performance testing setup, we demonstrate that our approach generates the target performance test cases for different programs more efficiently than a typical testing process and performs adaptively without access to source code and performance models.

show abstract

“…Another approach based on Hyper Heuristic methods [17] was proposed for combinatorial interaction testing to overcome the limitation of considering different testing strategies using simulated annealing and reinforcement learning. Reinforcement learning was also used for stress testing to find worst-case scenarios in a tool called PySE [19] and also for test data generation for unit testing using the branch criterion [18]. An approach [26] uses image recognition to distinguish part of the UI, to generate sequences via NLP seq2seq and to verify the values.…”

Section: Search-based Testingmentioning

confidence: 99%

Machine learning based test data generation for safety-critical software

Cegin¹

2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

View full text Add to dashboard Cite

PySE: Automatic Worst-Case Test Generation by Reinforcement Learning

Cited by 23 publications

References 28 publications

Automated Performance Testing Based on Active Deep Learning

Automated Performance Testing Based on Active Deep Learning

An autonomous performance testing framework using self-adaptive fuzzy reinforcement learning

Machine learning based test data generation for safety-critical software

Contact Info

Product

Resources

About