Cross-Dataset Design Discussion Mining

Abstract

Being able to identify software discussions that are primarily about design-which we call design mining-can improve documentation and maintenance of software systems. Existing design mining approaches have good classification performance using natural language processing (NLP) techniques, but the conclusion stability of these approaches is generally poor. A classifier trained on a given dataset of software projects has so far not worked well on different artifacts or different datasets. In this study, we replicate and synthesize these earlier results in a meta-analysis. We then apply recent work in transfer learning for NLP to the problem of design mining. However, for our datasets, these deep transfer learning classifiers perform no better than less complex classifiers. We conclude by discussing some reasons behind the transfer learning approach to design mining.

Conclusion Stability for Natural Language Based Mining of Design Discussions

“…Previous studies have introduced various vectorization techniques. In response to our previous study [15], we demonstrate how word embedding as a vectorization choice can improve the performance of the classifier. However, word embedding needs a reference model.…”

“…Early results from Robbes and Janes [23] reported on using ULMFiT [24] for sentiment analysis with some success. We also use the transfer NLP potential of ULMFiT, which we discuss in [15]. Robbes and Janes emphasized the importance of pretraining the learner on (potentially small) task-specific datasets.…”

On the relationship between design discussions and design quality: a case study of Apache projects

“…As manual classification is not a practical option to classify 1, 661, 922 discussions, we use machine learning techniques. We followed the protocol of Brunet et al [19] with some improvisations suggested by Mahadi et al [45].…”

“…Viviani et al [75] applied a classifier to automatically locate paragraphs in pull request discussions related to design. Mahadi et al [45] trained a classifier on the dataset created by Brunet et al [19] and tested it on the dataset of Viviani et al [75]. However, both of the dataset include discussions only from pull requests.…”

“…For example, Brunet et al [19] and Viviani et al [75,76] examined how design discussions are embedded in pull request comments and how it can be difficult for developers to piece together these discussions. Researchers have also developed techniques for detecting design discussions from a (single) communication channel using Machine Learning (ML) techniques [19,45,75,76].…”