Mining User Opinions to Support Requirement Engineering: An Empirical Study

Abstract

App reviews provide a rich source of user opinions that can support requirement engineering activities. Analysing them manually to find these opinions, however, is challenging due to their large quantity and noisy nature. To overcome the problem, automated approaches have been proposed for so-called opinion mining. These approaches facilitate the analysis by extracting features discussed in app reviews and identifying their associated sentiments. The effectiveness of these approaches has been evaluated using different methods and datasets. Unfortunately, replicating these studies to confirm their results and to provide benchmarks of different approaches is a challenging problem. We address the problem by extending previous evaluations and performing a comparison of these approaches. In this paper, we present an empirical study in which, we evaluated feature extraction and sentiment analysis approaches on the same dataset. The results show these approaches achieve lower effectiveness than reported originally, and raise an important question about their practical use.

Analysing app reviews for software engineering: a systematic literature review

“…To address the problem, 56 of the primary studies (31%) proposed approaches facilitating information extraction. Formally, information extraction is the task of extracting specific (pre-specified) information from the content of a review; this information may concern app features (Guzman and Maalej 2014;Johann et al 2017;Da ¸browski et al 2020), qualities (Groen et al 2017;Wang et al 2020b), problem reports and/or new feature requests (e.g., Iacob and Harrison 2013;Shams et al 2020), opinions about favored or unfavored features (e.g., Guzman and Maalej 2014;Gu and Kim 2015;Vu et al 2015a;Li et al 2017) as well as user stories (Guo and Singh 2020). Relevant information can be found at any location in the reviews.…”

“…Analysing these reviews can benefit a range of software engineering activities. For example, for requirements engineering, analyzing app reviews can help software engineers to elicit new requirements about app features that users desire (Johann et al 2017;Da ¸browski et al 2020); for testing, app reviews can help in finding bugs (Maalej and Nabil 2015;Iacob et al 2016;Shams et al 2020) and evaluating users' reactions to released beta versions of their apps AlSubaihin et al 2019); during product evolution, analysing app reviews may help in identifying and prioritizing change requests (Villarroel et al 2016;Gao et al 2018b;Da ¸browski et al 2020).…”

Holistic data-driven requirements elicitation in the big data era

“…The main motivation behind data-driven requirements elicitation is to exploit the availability of large amounts of digital data and thereby (i) consider a wider scope of sources of requirements, including potentially large and dispersed user bases, (ii) enable increased automation in processing relevant data by leveraging AI techniques based on, for instance, natural language processing and machine learning [6], (iii) facilitate continuous elicitation by rapidly considering the emergence of newly generated data, which helps to support software evolution through more frequent releases. Although the potential of digital data as sources of information for system requirements is becoming well-recognized, there are numerous challenges in processing this data effectively such that it can be fully exploited for development and evolution of enterprise software [7].…”

“…Being either human-generated, in the form of unstructured natural language, or machine generated, in the form of sensor data or computer logs, the data moreover tends to provide only implicit feedback on requirements; it is there-fore an inherent and fundamental challenge to transform the obtained raw data toward some canonical requirement format that is understandable to a software team and feasible to develop and implement, such as to the structured requirement template in the plan-based approach [2], or to the user story template in agile approaches [8]. Consequently, data-driven elicitation outcomes risk to be of low quality and practical use [8], or requiring a substantial manual effort [7,9]. Furthermore, the structure and semantics of the data obtained from different sources vary significantly, leading to numerous challenges concerning how to process and aggregate data into coherent requirements.…”

From Bag-of-Words to Pre-trained Neural Language Models: Improving Automatic Classification of App Reviews for Requirements Engineering

“…Popular apps usually receive hundreds of thousands or even millions of reviews and the manual analysis of so many reviews is an impracticable task. Several machine learning-based solutions have been proposed to solve this problem [Pagano and Maalej 2013, Guzman et al 2015, Wang et al 2018, Al Kilani et al 2019, Messaoud et al 2019, Dabrowski et al 2020, in which relevant information is extracted from the texts to automatically classify a review.…”