Domain Adaptation for Test Report Classification in Crowdsourced Testing

Wang, Junjie; Cui, Qiang; Wang, Song; Wang, Qing

doi:10.1109/icse-seip.2017.8

Cited by 25 publications

(20 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The above experimental results have shown that with the learning based techniques, the duplicate reports can be recognized with high accuracy, and these techniques can be utilized to facilitate the manual duplicate detection. According to existing work [45,47], the users prefer the actionable prediction outcomes, i.e., tagging each report with its predicted probability of being duplicates when presenting to the users for manual detection. In this way, for a candidate with higher probability, the users can treat it as the duplicate report without any consideration; on the contrary, the users can put more attention on those candidates with lower probabilities.…”

Section: Discussion and Threats To Validitymentioning

confidence: 99%

“…Finally, the output of CNN layer, the similarity output by similarity layer, and additional features are jointly input into the softmax layer for determining the duplicate status. To ensure the correctness of our implementation, we refer to two open source implementations on GitHub 45 .…”

Section: Deep Learning (Dl) Based Approachesmentioning

confidence: 99%

“…They designed strategies to dynamically select the most risky and diversified test report for inspection in each iteration. Wang et al [44,45,47] proposed approaches to automatically classify crowdtesting reports. Their approaches can overcome the different data distribution among different software domains, and attain good classification results.…”

Section: Related Workmentioning

confidence: 99%

“…A typical report describes how the test was performed and what happened during the test. In order to attract workers, testing tasks are often financially compensated; thus workers may submit thousands of test reports due to financial incentive and other motivations [14,15,44,45,47,50]. Given the invisibility and unpredictability of distributed crowdtesting processes, there are large number of duplicate reports.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Quest for the Golden Approach

Huang

Wang

et al. 2020

Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

Self Cite

View full text Add to dashboard Cite

Background: Given the invisibility and unpredictability of distributed crowdtesting processes, there is a large number of duplicate reports, and detecting these duplicate reports is an important task to help save testing effort. Although, many approaches have been proposed to automatically detect the duplicates, the comparison among them and the practical guidelines to adopt these approaches in crowdtesting remain vague. Aims: We aim at conducting the first experimental evaluation of the commonly-used and state-of-the-art approaches for duplicate detection in crowdtesting reports, and exploring which is the golden approach. Method: We begin with a systematic review of approaches for duplicate detection, and select ten state-of-the-art approaches for our experimental evaluation. We conduct duplicate detection with each approach on 414 crowdtesting projects with 59,289 reports collected from one of the largest crowdtesting platforms. Results: Machine learning based approach, i.e., ML-REP, and deep learning based approach, i.e., DL-BiMPM, are the best two approaches for duplicate reports detection in crowdtesting, while the later one is more sensitive to the size of training data and more time-consuming for model training and prediction. Conclusions: This paper provides new insights and guidelines to select appropriate duplicate detection techniques for duplicate crowdtesting reports detection. CCS CONCEPTS • Software and its engineering → Software testing and debugging.

show abstract

Section: Discussion and Threats To Validitymentioning

confidence: 99%

Section: Deep Learning (Dl) Based Approachesmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Quest for the Golden Approach

Huang

Wang

et al. 2020

Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the area of software development (Table ), we found publications related to various stages of the development process, and, especially, to the software maintenance stage, where developers receive bugs and reviews regarding their application and they have to fix the software or improve its capabilities. Specifically, we found applications related to predicting fidelity of mobile application reviews (Kong et al, ), source code summarization (Nazar et al, ), which facilitates understanding the code that needs to be maintained, and classification of bug reports (Hernández‐González et al, ; Wang et al, ), which helps to ensure that bugs are tackled by the best person in the team. There are also applications that use crowdsourcing to improve the performance of its estimates of occupancy in parking lots (Davami & Sukthankar, ) and of analysis of malware (Du et al, ).…”

Section: Publication Areasmentioning

confidence: 99%

Machine learning from crowds: A systematic review of its applications

Rodrigo

Aledo

Gámez

2018

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Crowdsourcing opens the door to solving a wide variety of problems that previously were unfeasible in the field of machine learning, allowing us to obtain relatively low cost labeled data in a small amount of time. However, due to the uncertain quality of labelers, the data to deal with are sometimes unreliable, forcing practitioners to collect information redundantly, which poses new challenges in the field. Despite these difficulties, many applications of machine learning using crowdsourced data have recently been published that achieved state of the art results in relevant problems. We have analyzed these applications following a systematic methodology, classifying them into different fields of study, highlighting several of their characteristics and showing the recent interest in the use of crowdsourcing for machine learning. We also identify several exciting research lines based on the problems that remain unsolved to foster future research in this field.

show abstract

False negative of defects estimation in crowdsourced testing

Shi

Zhu

et al. 2022

J Software Evolu Process

View full text Add to dashboard Cite

Estimating the population of defects is a key reference for completion evaluation in crowdsourced testing. Current studies commonly use the biological capture–recapture model (CRC model) to estimate the population of defects in crowdsourced tests with good results. However, the lack of consideration of false negatives for defects in existing studies leads to imprecise estimation of completion in crowdsourced testing, which in turn influences decision support during crowdsourced testing. This study analyzes the basic assumptions of the CRC model in biology and maps them to the field of crowdsourced testing, improves the Lincoln–Peterson estimator and sample coverage estimator utilizing false negative assumptions, and compares them with original estimators in 29 real crowdsourced testing projects. The experimental results show that our improved estimators can effectively reduce the estimation relative error of the population of defects caused by false negatives.

show abstract

Domain Adaptation for Test Report Classification in Crowdsourced Testing

Cited by 25 publications

References 21 publications

Quest for the Golden Approach

Quest for the Golden Approach

Machine learning from crowds: A systematic review of its applications

False negative of defects estimation in crowdsourced testing

Contact Info

Product

Resources

About