Quest for the Golden Approach

Abstract

Background: Given the invisibility and unpredictability of distributed crowdtesting processes, there is a large number of duplicate reports, and detecting these duplicate reports is an important task to help save testing effort. Although, many approaches have been proposed to automatically detect the duplicates, the comparison among them and the practical guidelines to adopt these approaches in crowdtesting remain vague. Aims: We aim at conducting the first experimental evaluation of the commonly-used and state-of-the-art approaches for duplicate detection in crowdtesting reports, and exploring which is the golden approach. Method: We begin with a systematic review of approaches for duplicate detection, and select ten state-of-the-art approaches for our experimental evaluation. We conduct duplicate detection with each approach on 414 crowdtesting projects with 59,289 reports collected from one of the largest crowdtesting platforms. Results: Machine learning based approach, i.e., ML-REP, and deep learning based approach, i.e., DL-BiMPM, are the best two approaches for duplicate reports detection in crowdtesting, while the later one is more sensitive to the size of training data and more time-consuming for model training and prediction. Conclusions: This paper provides new insights and guidelines to select appropriate duplicate detection techniques for duplicate crowdtesting reports detection. CCS CONCEPTS • Software and its engineering → Software testing and debugging.

Severity-Oriented Multi-Objective Crowdsourced Test Reports Prioritization

“…In addition to the research mentioned above on methodological proposals, Huang et al [29] conducted the first experimental evaluation of 10 duplicate crowdsourced test report detection methods to explore which is the golden method. The results show that ML-REP, a machine learning-based approach, and DL-BiMPM, a deep learningbased method, are the best; however, the latter is more sensitive to training data quantity and takes longer to train and predict.…”

Optimizing Prioritization of Crowdsourced Test Reports of Web Applications through Image-to-Text Conversion

“…Cooper et al [23] proposed a video-based duplicate detection method named TANGO that combines computer vision, optical character recognition, and text retrieval methods. Huang et al [24] conducted an empirical evaluation of 10 commonly used state-of-the-art duplicate detection methods. They found that machine learning-based methods, i.e., ML-REP, and deep learning-based methods, i.e., DL-BiMPM, are the best two methods; the latter is sensitive to the size of the training data.…”