Automatic Identification of Assumptions from the Hibernate Developer Mailing List

Abstract

During the software development life cycle, assumptions are an important type of software development knowledge that can be extracted from textual artifacts. Analyzing assumptions can help to, for example, comprehend software design and further facilitate software maintenance. Manual identification of assumptions by stakeholders is rather timeconsuming, especially when analyzing a large dataset of textual artifacts. To address this problem, one promising way is to use automatic techniques for assumption identification. In this study, we conducted an experiment to evaluate the performance of existing machine learning classification algorithms for automatic assumption identification, through a dataset extracted from the Hibernate developer mailing list. The dataset is composed of 400 "Assumption" sentences and 400 "Non-Assumption" sentences. Seven classifiers using different machine learning algorithms were selected and evaluated. The experiment results show that the SVM algorithm achieved the best performance (with a precision of 0.829, a recall of 0.812, and an F1-score of 0.819). Additionally, according to the ROC curves and related AUC values, the SVM-based classifier comparatively performed better than other classifiers for the binary classification of assumptions.

Analyst’s Perception on the Use of AI-based Tools in the Software Development Life Cycle

“…Related to the analysis phase, ML techniques can help with software requirement specifications (Akshatha Nayak et al 2022;Quba et al 2021) and can also be used to predict software vulnerabilities (Imtiaz et al, 2021). Related to the design phase, various AI techniques can be used, such as ML which is used to assist in predicting software bug (Delphine Immaculate et al 2019) and automate the assumption identification process (Li et al 2019), NLP to create DFDs (Cheema et al 2023) and used for voice-driven modeling software (Black et al 2021), Artificial Neural Network (ANN) for software bug prediction (P and Kambli 2020), as well as the use of tools based on intelligence decision support systems used in risk management software (Asif and Ahmed 2020). Related to the fourth phase of SDLC, implementation, ML techniques can be u can be applied for a variety of purposes.…”

Automatic Identification and Extraction of Assumptions on GitHub

“…In their study, they identified and extracted 832 assumptions. Li et al developed a machine learning approach [16] to identify and classify assumptions based on the dataset constructed by Xiong et al [15], which can read the data (i.e., sentences) from the dataset (i.e., a .CSV file), preprocess the data (e.g., using NLTK and Word2Vec), train classifiers (e.g., Perception, Logistic Regression, and Support Vector Machines), and evaluate the trained classifiers (e.g., precision, recall, and F1-score). However, their approach is not specifically developed for PAs and SCAs and cannot mine assumptions from other sources (e.g., GitHub repositories).…”

A Machine Learning Based Ensemble Method for Automatic Multiclass Classification of Decisions

“…Besides the studies that focus on decisions, many studies have also focused on automatically mining artifacts that are highly related to decisions. Li et al used Word2Vec to extract features and trained seven machine learning classifiers for automatically identifying assumptions from an OSS developer mailing list [18]. Their experiment results show that the automatic approach can effectively identify assumptions by using the SVM algorithm with a precision of 0.829, a recall of 0.812, and an F1-score of 0.819.…”

“…LR achieves the best performance when identifying codesmell discussions from SO [35]. SVM gets the best classification result when identifying assumptions and decisions from the Hibernate developer mailing list [18,19]. Therefore, four widely used machine learning classifiers (i.e., NB, LR, SVM, and RF) are used as base classifiers in our experiment to seek the best configuration to classify decisions.…”