What is wrong with topic modeling? And how to fix it using search-based software engineering

Abstract

Context: Topic modeling finds human-readable structures in unstructured textual data. A widely used topic modeling technique is Latent Dirichlet allocation. When running on different datasets, LDA suffers from "order effects", i.e., different topics are generated if the order of training data is shuffled. Such order effects introduce a systematic error for any study. This error can relate to misleading results; specifically, inaccurate topic descriptions and a reduction in the efficacy of text mining classification results. Objective: To provide a method in which distributions generated by LDA are more stable and can be used for further analysis. Method: We use LDADE, a search-based software engineering tool which uses Differential Evolution (DE) to tune the LDA's parameters. LDADE is evaluated on data from a programmer information exchange site (Stackoverflow), title and abstract text of thousands of Software Engineering (SE) papers, and software defect reports from NASA. Results were collected across different implementations of LDA (Python+Scikit-Learn, Scala+Spark) across Linux platform and for different kinds of LDAs (VEM, Gibbs sampling). Results were scored via topic stability and text mining classification accuracy. Results: In all treatments: (i) standard LDA exhibits very large topic instability; (ii) LDADE's tunings dramatically reduce cluster instability; (iii) LDADE also leads to improved performances for supervised as well as unsupervised learning. Conclusion: Due to topic instability, using standard LDA with its "off-the-shelf" settings should now be depreciated. Also, in future, we should require SE papers that use LDA to test and (if needed) mitigate LDA topic instability. Finally, LDADE is a candidate technology for effectively and efficiently reducing that instability.

Guest editorial for special section on success and failure in software engineering

“…We used a fixed number of clusters (k = 10) to offer an easily understandable overview of the area and allow a sample of papers for the qualitative analysis to remain small, rather than generating large number of fine-grained clusters. We used the genetic algorithm Differential Evolution to tune LDA hyperparameters alpha and beta as suggested by Agrawal et al (2016).…”

Analysis and Detection of Information Types of Open Source Software Issue Discussions

“…Our initial exploration suggests that text-based unsupervised methods such as topic modeling and clustering are not effective in detecting the information types. The performances of such methods are sensitive to the parameter settings [2] and are highly dependant on the distribution of the terms which can not be generalized across domains [6]. Consequently, in this section, we explore the possibility of utilizing supervised techniques to detect information types of sentences in issue Fig.…”

Semantic Source Code Models Using Identifier Embeddings

“…Source code embeddings can contribute in inferring semantic inconsistencies and assist tasks such as semantic bug localization and recommendations for semantic bug fix. Robust Topic Modeling: Agrawal et al [38] present a comprehensive review of topic modeling studies in software engineering. Following the paradigm of natural language word embeddings, pretrained source code embeddings provide background knowledge that can further enhance existing methods.…”