Automatic Web Testing Using Curiosity-Driven Reinforcement Learning

Yan, Zheng; Liu, Yi; Xie, Xiaofei; Liu, Yepang; Ма, Лей; Hao, Jianye; Liu, Yang

doi:10.1109/icse43902.2021.00048

Cited by 37 publications

(14 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For undocumented APIs, a possible solution is to infer their related schemes in an online manner by analyzing the feedback of the testing process. Another possible solution is to use client-side analysis techniques [31,32,49] to generate traffic between clients and servers and analyze the traffic to infer the correct usage of the undocumented APIs. We leave the support of undocumented APIs as future work.…”

Section: Threats To Validitymentioning

confidence: 99%

“…Model-based testing techniques. Besides the techniques for RESTful API and SOAP service testing, model-based testing techniques [13,14,36,48,49] are also related to Morest. In general, these techniques use models, say a finite-state machine [29], to describe the system-under-test and generate tests by covering different states in the model [45].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Morest: Model-based RESTful API Testing with Execution Feedback

Liu¹,

Li²,

Deng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

RESTful APIs are arguably the most popular endpoints for accessing Web services. Blackbox testing is one of the emerging techniques for ensuring the reliability of RESTful APIs. The major challenge in testing RESTful APIs is the need for correct sequences of API operation calls for in-depth testing. To build meaningful operation call sequences, researchers have proposed techniques to learn and utilize the API dependencies based on OpenAPI specifications. However, these techniques either lack the overall awareness of how all the APIs are connected or the flexibility of adaptively fixing the learned knowledge.In this paper, we propose Morest, a model-based RESTful API testing technique that builds and maintains a dynamically updating RESTful-service Property Graph (RPG) to model the behaviors of RESTful-services and guide the call sequence generation. We empirically evaluated Morest and the results demonstrate that Morest can successfully request an average of 152.66%-232.45% more API operations, cover 26.16%-103.24% more lines of code, and detect 40.64%-215.94% more bugs than state-of-the-art techniques. In total, we applied Morest to 6 real-world projects and found 44 bugs (13 of them cannot be detected by existing approaches). Specifically, 2 of the confirmed bugs are from Bitbucket, a famous code management service with more than 6 million users.

show abstract

Section: Threats To Validitymentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Morest: Model-based RESTful API Testing with Execution Feedback

Liu¹,

Li²,

Deng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…usage specifications [47,48], unique code functions called [50], a curiosity factorfavoring exploration of new elements [51,54]-coverage of interaction methods (e.g. click, drag) [46], and avoidance of navigation loops [44].…”

Section: Gui Test Generationmentioning

confidence: 99%

“…If a function is simple, there is likely little need for a predictive model in the first place. Several recent studies feature more thorough evaluations (e.g., [39,5,54]), even on industrial systems (e.g., [32,16]). However, it largely remains to be seen whether the proposed techniques can be used on real-world production code.…”

Section: Rq6: Limitations and Open Challengesmentioning

confidence: 99%

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Fontes¹,

Gay²

2022

Preprint

View full text Add to dashboard Cite

Context: Machine learning (ML) may enable effective automated test generation.Objectives: We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges.Methods: We perform a systematic literature review on a sample of 97 publications.Results: ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property-based, and expected output oracles. Supervised learning-often based on neural networks-and reinforcement learning-often based on Q-learning-are common, and some publications also employ unsupervised or semi-supervised learning. (Semi-/Un-)Supervised approaches are evaluated using both traditional testing metrics and ML-related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. Conclusion:Work-to-date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed-and how they are applied-benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.

show abstract

“…Deep learning (DL) has achieved great success in various domains, such as computer vision [29], natural language processing [26], code understanding [30], and autonomous driving [18]. Due to the outstanding performance of deep neural networks (DNNs), researchers from the software engineering (SE) community have attempted to apply DNNs for diverse SE tasks, such as source code processing [7,30], automatic software testing [41,42], and GUI designs [40]. Generally, for developers, compared with designing a new DNN architecture that requires tremendous DL knowledge, reusing publicly available models is an efficient and popular strategy to avoid preparing everything from scratch.…”

Section: Introductionmentioning

confidence: 99%

Labeling-Free Comparison Testing of Deep Learning Models

Guo¹,

Hu²,

Cordy³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Various deep neural networks (DNNs) are developed and reported for their tremendous success in multiple domains. Given a specific task, developers can collect massive DNNs from public sources for efficient reusing and avoid redundant work from scratch. However, testing the performance (e.g., accuracy and robustness) of multiple DNNs and giving a reasonable recommendation that which model should be used is challenging regarding the scarcity of labeled data and demand of domain expertise. Existing testing approaches are mainly selection-based where after sampling, a few of the test data are labeled to discriminate DNNs. Therefore, due to the randomness of sampling, the performance ranking is not deterministic. In this paper, we propose a labeling-free comparison testing approach to overcome the limitations of labeling effort and sampling randomness. The main idea is to learn a Bayesian model to infer the models' specialty only based on predicted labels. To evaluate the effectiveness of our approach, we undertook exhaustive experiments on 9 benchmark datasets spanning in the domains of image, text, and source code, and 165 DNNs. In addition to accuracy, we consider the robustness against synthetic and natural distribution shifts. The experimental results demonstrate that the performance of existing approaches degrades under distribution shifts. Our approach outperforms the baseline methods by up to 0.74 and 0.53 on Spearman's correlation and Kendall's 𝜏, respectively, regardless of the dataset and distribution shift. Additionally, we investigated the impact of model quality (accuracy and robustness) and diversity (standard deviation of the quality) on the testing effectiveness and observe that there is a higher chance of a good result when the quality is over 50% and the diversity is larger than 18%.

show abstract

Automatic Web Testing Using Curiosity-Driven Reinforcement Learning

Cited by 37 publications

References 44 publications

Morest: Model-based RESTful API Testing with Execution Feedback

Morest: Model-based RESTful API Testing with Execution Feedback

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Labeling-Free Comparison Testing of Deep Learning Models

Contact Info

Product

Resources

About