On Using Active Learning and Self-training when Mining Performance Discussions on Stack Overflow

Abstract

Abundant data is the key to successful machine learning. However, supervised learning requires annotated data that are often hard to obtain. In a classification task with limited resources, Active Learning (AL) promises to guide annotators to examples that bring the most value for a classifier. AL can be successfully combined with self-training, i.e., extending a training set with the unlabelled examples for which a classifier is the most certain. We report our experiences on using AL in a systematic manner to train an SVM classifier for Stack Overflow posts discussing performance of software components. We show that the training examples deemed as the most valuable to the classifier are also the most difficult for humans to annotate. Despite carefully evolved annotation criteria, we report low inter-rater agreement, but we also propose mitigation strategies. Finally, based on one annotator's work, we show that self-training can improve the classification accuracy. We conclude the paper by discussing implication for future text miners aspiring to use AL and self-training.

Enabling Image Recognition on Constrained Devices Using Neural Network Pruning and a CycleGAN

“…The obvious first step is to extend the dataset used for both the classification model and the CycleGAN. As the data labelling is labor-intensive, we plan to rely on our previous experience in active learning to focus annotation effort for maximum return on investment [4]. With more data, we can train the classifier to predict additional classes, including input related to emergency response.…”

LeSSA: A Unified Framework based on Lexicons and Semi-Supervised Learning Approaches for Textual Sentiment Classification

“…Active learning techniques select the most informative unlabeled examples to predict their label and include them in the training set [63]. Many researchers have successfully combined active learning with self-training to decrease the human labeling struggle and enhance the classification performance [33,64]. Motivated by the existing research [33], we integrate active learning with selftraining to select the most informative and high confident level examples.…”

Practitioner Evaluations on Software Testing Tools

“…In software engineering (SE), there are studies reporting low values for expert agreement/reliability using Kirppendorff's alpha and/or ICC, by e.g., Borg et al [4], Anvaari et al [1] and Kitchenham et al [27]. Evaluations depend on the interpretation of a construct under study, i.e., include some degree of subjectivity [5,47].…”

“…The values α ≥ 0.800 are suggested for drawing reliable conclusions while values 0.667 ≤ α < 0.800 are claimed for tentative conclusions only [29]. We used the R-function kripp.alpha 4 to measure the level of agreement among the respondents (raters) on the criteria (subjects) of the top 6 most evaluated tools. We considered the level of measurement for the data to be ratio, since the possible values (from 0 to 10 at intervals of 0.5, i.e., 21 levels) were ordered units having the same difference and an absolute zero.…”