Classifying Multilingual User Feedback using Traditional Machine Learning and Deep Learning

Abstract

With the rise of social media like Twitter and of software distribution platforms like app stores, users got various ways to express their opinion about software products. Popular software vendors get user feedback thousandfold per day. Research has shown that such feedback contains valuable information for software development teams such as problem reports or feature and support inquires. Since the manual analysis of user feedback is cumbersome and hard to manage many researchers and tool vendors suggested to use automated analyses based on traditional supervised machine learning approaches. In this work, we compare the results of traditional machine learning and deep learning in classifying user feedback in English and Italian into problem reports, inquiries, and irrelevant. Our results show that using traditional machine learning, we can still achieve comparable results to deep learning, although we collected thousands of labels.

Data-Driven Requirements Elicitation: A Systematic Literature Review

“…Finally, a very important direction relates to the proposals for enabling context-awareness to capture requirements that changes dynamically over time. [73] 2016 International Conference on Research Challenges in Information Science (RCIS) C33 Bakar et al [62] 2015 International Conference on Information Science and Security (ICISS) C34 Maalej and Nabil [39] 2015 IEEE International Requirements Engineering Conference (RE) C35 Panichella et al [45] 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME) C36 Takahashi et al [100] 2015 International Conference on Software Engineering & Knowledge Engineering (SEKE) C37 Sun and Peng [53] 2015 Asia Pacific Requirements Engineering Symposium (APRES) C38 Guzman and Maalej [40] 2014 IEEE International Requirements Engineering Conference (RE) C39 Jiang et al [41] 2014 Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD) C40 Zhang et al [60] 2014 International Conference on Enterprise Systems (ES) C41 Carreño and Winbladh [52] 2013 International Conference on Software Engineering (ICSE) W1 Stanik et al [65] 2019 International Requirements Engineering Conference Workshops (REW) W2 Do and Bhowmik [36] 2018 ACM SIGSOFT International Workshop on Automated Specification Inference W3 Abad et al [44] 2017 International Requirements Engineering Conference Workshops (REW) W4 Bakiu and Guzman [55] 2017 International Requirements Engineering Conference Workshops (REW) W5 Deocadez et al [43] 2017 International Requirements Engineering Conference Workshops (REW) W6 Jha and Mahmoud [50] 2017 REFSQ Workshops W7 Portugal et al [86] 2015 IEEE Workshop on Just-In-Time Requirements Engineering (JITRE) S1…”

“…• Microblog data from twitter, Facebook, and Weibo were used for automated requirements elicitation. Of total eleven studies that used twitter, four studies extracted only texts [64][65][66][67], while the rest extracted additional metadata [68][69][70][71][72][73][74]. The metadata include the number of retweets, likes, lexically similar tweets (i.e., duplicates), twitter followers and friends (i.e., social rank), replies to tweets, as well as hashtags, handles (i.e., indicated by an @ appended with a username), and demographic data of the person who tweeted.…”

“…However, those two types of datasets were used independently without being integrated to evaluate their proposed elicitation process. Moreover, Stanik et al [65] used three datasets: app reviews, tweets written in English, and tweets written in Italian. On the other hand, Johann et al [94] integrated both app reviews and descriptions to provide information on which app features are or are not actually reviewed.…”

“…All the studies included in the review, except one utilizing user feedback in both English and Italian [65], focus on eliciting English requirements. Thus, our results cannot be generalizable to requirements elicitation in other languages.…”

Coding and Classifying Knowledge Exchange on Social Media: a Comparative Analysis of the #Twitterstorians and AskHistorians Communities

“…First, we applied a collection of six base sub-models: Random Forest (RF), Support Vector Machine (SVM), Gradient Boosting Machine (GBM), Naïve-Bayes, Logistic Regression (LR), and Tree Bagging (TB). These models are commonly used in relevant literature on classification of Twitter text (Guzman et al 2017;Stanik et al 2019;Williams and Mahmoud 2017). We eliminated highly correlated base models (correlation values higher than 0.75).…”