Can app changelogs improve requirements classification from app reviews?

Abstract

Background] Recent research on mining app reviews for software evolution indicated that the elicitation and analysis of user requirements can benefit from supplementing user reviews by data from other sources. However, only a few studies reported results of leveraging app changelogs together with app reviews.[Aims] Motivated by those findings, this exploratory experimental study looks into the role of app changelogs in the classification of requirements derived from app reviews. We aim at understanding if the use of app changelogs can lead to more accurate identification and classification of functional and non-functional requirements from app reviews. We also want to know which classification technique works better in this context. [Method]We did a case study on the effect of app changelogs on automatic classification of app reviews. Specifically, manual labeling, text preprocessing, and four supervised machine learning algorithms were applied to a series of experiments, varying in the number of app changelogs in the experimental data. [Results] We compared the accuracy of requirements classification from app reviews, by training the four classifiers with varying combinations of app reviews and changelogs. Among the four algorithms, Naïve Bayes was found to be more accurate for categorizing app reviews.[Conclusions] The results show that official app changelogs did not contribute to more accurate identification and classification of requirements from app reviews. In addition, Naïve Bayes seems to be more suitable for our further research on this topic. CCS CONCEPTS• Software and its engineering → Requirements analysis

Augmenting App Review with App Changelogs: An Approach for App Review Classification

“…The experimental data collected and manually labeled in our previous work [13] will be reused to evaluate the performance and AUC-AC. The dataset includes 6000 app review sentences of three apps (one from Apple App Store and two from Google Play) and 2024 app changes filtered from 2005 official changelogs of 30 apps (3 ×10 in Apple App Store).…”

“…The dataset includes 6000 app review sentences of three apps (one from Apple App Store and two from Google Play) and 2024 app changes filtered from 2005 official changelogs of 30 apps (3 ×10 in Apple App Store). As described in [13], these app review sentences and changes were labeled with six types of requirements, including four types of NFRs defined in ISO 25010 [11] (i.e. Usability, Reliability, Portability and Performance), FR, and 'Others'-the type referring to those review sentences and app changes that fit neither FRs nor the four NFRs listed above.…”

“…According to the experimental results in [1,9,13], Naïve Bayes has been reported to outperform other machine learning algorithms in the automatic classification of app reviews. Therefore, this work adopted Naïve Bayes as the classification technique to categorize FR and NFRs from app reviews.…”

“…Our work reused the dataset in [13]. All the app reviews and changelogs in this dataset were analyzed by three coders independently, on the premise that they had a consistent understanding on different types of requirements, especially on NFR types defined in ISO 25010.…”

“…All the app reviews and changelogs in this dataset were analyzed by three coders independently, on the premise that they had a consistent understanding on different types of requirements, especially on NFR types defined in ISO 25010. As indicated in [13], we believe this threat to construct validity is partially mitigated by following the aforementioned labelling process.…”