Towards Effectively Test Report Classification to Assist Crowdsourced Testing

Wang, Junjie; Cui, Qiang; Qing, Wang; Wang, Song

doi:10.1145/2961111.2962584

Cited by 44 publications

(34 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally terms except verbs and nouns are removed. Filtering meaningless terms like existing work [15], we obtain a vocabulary of technical terms.…”

Section: Baidu Crowdtest Datasetmentioning

confidence: 99%

“…All these studies use crowdsourced testing to solve the problems in traditional software testing activities. There are studies focusing on solving the new encountered problem in crowdsourced testing, e.g., crowdsourced reports prioritization [5] and crowdsourced reports classification [15,16]. Our approach is also to solve the new encountered and important problem in crowdsourced testing.…”

Section: Rq2: Is Each Of the Three Objectives Necessary In Moose?mentioning

confidence: 99%

“…In recent years, it has received much attention from the research community [15,16], and there are many successful crowdsourced testing platforms in industry, e.g., uTest 1 , TestBirds 2 , and Pay4Bugs 3 . Different from traditional testing, crowdsourced testing entrusts test tasks to crowd workers, who are available on Internet and located in different places.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Multi-Objective Crowd Worker Selection in Crowdsourced Testing

Cui

Wang

et al. 2017

International Conferences on Software Engineering and Knowledge Engineering

Self Cite

View full text Add to dashboard Cite

Abstract-Crowdsourced testing is an emerging trend in software testing, which relies on crowd workers to accomplish test tasks. Typically, a crowdsourced testing task aims to detect as many bugs as possible within a limited budget. For a specific test task, not all crowd workers are qualified to perform it, and different test tasks require crowd workers to have different experiences, domain knowledge, etc. Inappropriate workers may miss true bugs, introduce false bugs, or report duplicated bugs, which could not only decrease the quality of test outcomes, but also increase the cost of hiring workers. Thus, how to select the appropriate crowd workers for specific test tasks is a challenge in crowdsourced testing.This paper proposes a Multi-Objective crowd wOrker SElection approach (MOOSE), which includes three objectives: maximizing the coverage of test requirement, minimizing the cost, and maximizing bug-detection experience of the selected crowd workers. Specifically, MOOSE leverages NSGA-II, a widely used multi-objective evolutionary algorithm, to optimize the three objectives when selecting workers. We evaluate MOOSE on 42 test tasks (involve 844 crowd workers and 3,984 test reports) from one of the largest crowdsourced testing platforms in China, and the experimental results show MOOSE could improve the best baseline by 17% on average in bug detection rate.

show abstract

“…Finally terms except verbs and nouns are removed. Filtering meaningless terms like existing work [15], we obtain a vocabulary of technical terms.…”

Section: Baidu Crowdtest Datasetmentioning

confidence: 99%

Section: Rq2: Is Each Of the Three Objectives Necessary In Moose?mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Objective Crowd Worker Selection in Crowdsourced Testing

Cui

Wang

et al. 2017

International Conferences on Software Engineering and Knowledge Engineering

Self Cite

View full text Add to dashboard Cite

show abstract

“…They proposed prioritization approaches based on text descriptions, screenshot images, and a combination of both sources of information. Wang et al [10] proposed a cluster-based classification approach for effectively classifying crowdsourced reports when plentiful training data are available. However, sufficient training data often are not available.…”

Section: Crowdsourced Software Testingmentioning

confidence: 99%

“…They designed strategies for dynamically selecting the riskiest and most diversified test reports for inspection in each iteration. Wang et al [10,11] proposed a cluster-based classification approach for effective classification of crowdsourced reports that addresses the local bias problem. Unfortunately, the Android bug reports do not have the severity labels for use as training data, and these approaches often require users to manually label a large number of training data, which is both time-consuming and labor-intensive in practice.…”

Section: Introductionmentioning

confidence: 99%

Using Knowledge Transfer and Rough Set to Predict the Severity of Android Test Reports via Text Mining

Guo

Chen

2017

Symmetry

View full text Add to dashboard Cite

Abstract:Crowdsourcing is an appealing and economic solution to software application testing because of its ability to reach a large international audience. Meanwhile, crowdsourced testing could have brought a lot of bug reports. Thus, in crowdsourced software testing, the inspection of a large number of test reports is an enormous but essential software maintenance task. Therefore, automatic prediction of the severity of crowdsourced test reports is important because of their high numbers and large proportion of noise. Most existing approaches to this problem utilize supervised machine learning techniques, which often require users to manually label a large number of training data. However, Android test reports are not labeled with their severity level, and manual labeling is time-consuming and labor-intensive. To address the above problems, we propose a Knowledge Transfer Classification (KTC) approach based on text mining and machine learning methods to predict the severity of test reports. Our approach obtains training data from bug repositories and uses knowledge transfer to predict the severity of Android test reports. In addition, our approach uses an Importance Degree Reduction (IDR) strategy based on rough set to extract characteristic keywords to obtain more accurate reduction results. The results of several experiments indicate that our approach is beneficial for predicting the severity of android test reports.

show abstract

Mobile crowdsourced test report prioritization based on text and image understanding

Tong

Liu

et al. 2023

J Software Evolu Process

View full text Add to dashboard Cite

In the process of mobile crowdsourced testing, a large number of test reports are generated, which often consist of short text and rich image information. One of the critical issues is how to review these reports efficiently. Researchers have recently proposed clustering, classification, and prioritization techniques to solve this problem. However, existing studies directly use related technologies to text description and segment the text content without further understanding of the text content. By deeply digging into the text information and distinguishing it according to its actual meaning, the sentences described in the text can be categorized into two types: Describing abnormal system behavior and describing reproduction steps. This paper proposes a mobile crowdsourced test report prioritization technique to improve performance. First, we use a TextCNN trained on large‐scale projects to distinguish the text descriptions of reports, then extract features from the text and screenshot information, respectively. Then we apply a clustering technique to gather similar reports. Finally, the inspection order is sampled from the clustering results. To validate our approach, we conduct experiments on six industrial crowdsourced projects. The results show that our method can detect all bugs faster in a limited time than existing prioritization methods, which can improve the bug reports review efficiency.

show abstract

Towards Effectively Test Report Classification to Assist Crowdsourced Testing

Cited by 44 publications

References 24 publications

Multi-Objective Crowd Worker Selection in Crowdsourced Testing

Multi-Objective Crowd Worker Selection in Crowdsourced Testing

Using Knowledge Transfer and Rough Set to Predict the Severity of Android Test Reports via Text Mining

Mobile crowdsourced test report prioritization based on text and image understanding

Contact Info

Product

Resources

About