Assessment of off-the-shelf SE-specific sentiment analysis tools: An extended replication study

Novielli, Nicole; Calefato, Fabio; Lanubile, Filippo; Serebrenik, Alexander

doi:10.1007/s10664-021-09960-w

Cited by 21 publications

(19 citation statements)

References 43 publications

(89 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Inspired by the recent advances in deep learning, Biswas et al [11] and Zhang et al [10] also investigated the application of pretrained language models, BERT in particular, in sentiment classification; they found that the BERT sentiment classifier out-performed several other state-of-the-art sentiment analysis techniques on a Stack Overflow dataset. A recent replication study found that Senti4SD has achieved similar performance than BERT-based techniques; the former also out-performed the latter on a GitHub discussion dataset [37].…”

Section: Automated Detection Of Sentiments and Tones In Textmentioning

confidence: 96%

The Impacts of Sentiments and Tones in Community-Generated Issue Discussions

Sanei¹,

Cheng²,

Adams³

2021

Preprint

View full text Add to dashboard Cite

The diverse community members who contribute to the discussions on issue tracking systems of open-source software projects often exhibit complex affective states such as sentiments and tones. These affective states can significantly influence the effectiveness of the issue discussions in elaborating the initial ideas into actionable tasks that the development teams need to address. In this paper, we present an extended empirical study to investigate the impacts of sentiments and tones in community-generated issue discussions. We created and validated a large dataset of sentiments and tones in the issues posts and comments created by diverse community members in three popular open source projects. Our analysis results drew a complex picture of the relationships between, on the one hand, the sentiments and tones in the issue discussions, and on the other hand, various discussion and development-related measures such as the discussion length and the issue resolution time. We also found that when factors such as the issue poster roles and the issue types were controlled, sentiments and tones had varied associations with the measures. Insights gained from these findings can support open source community members in making and moderating effective issue discussions and guide the design of tools to better support community engagement.

show abstract

Section: Automated Detection Of Sentiments and Tones In Textmentioning

confidence: 96%

The Impacts of Sentiments and Tones in Community-Generated Issue Discussions

Sanei¹,

Cheng²,

Adams³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The second type of task should have shorter text, to evaluate how good the model is for short texts. The third task should be sentiment mining, as a general domain use case that is already well understood in the SE domain [15], incl. strong, already existing results that general domain transformer models are the current state-ofthe-art for sentiment mining in the SE domain [14].…”

Section: Fine-tuned Prediction Tasksmentioning

confidence: 99%

“…Additionally, we also consider sentiment mining as fine-tuning task, a task where it was already shown that general domain transformers perform well [14]. Since sentiment mining possibly also works on SE domain data without SE domain knowledge [15], the evaluation of this tasks allows us to understand if even tasks that are not SE specific can be improved with SE domain transformers.…”

Section: Introductionmentioning

confidence: 99%

On the validity of pre-trained transformers for natural language processing in the software engineering domain

Mosel¹,

Trautsch²,

Herbold³

2021

Preprint

View full text Add to dashboard Cite

Transformers are the current state-of-the-art of natural language processing in many domains and are using traction within software engineering research as well. Such models are pre-trained on large amounts of data, usually from the general domain. However, we only have a limited understanding regarding the validity of transformers within the software engineering domain, i.e., how good such models are at understanding words and sentences within a software engineering context and how this improves the state-of-the-art. Within this article, we shed light on this complex, but crucial issue. We compare BERT transformer models trained with software engineering data with transformers based on general domain data in multiple dimensions: their vocabulary, their ability to understand which words are missing, and their performance in classification tasks. Our results show that for tasks that require understanding of the software engineering context, pre-training with software engineering data is valuable, while general domain models are sufficient for general language understanding, also within the software engineering domain.

show abstract

“…However, our results do not contain enough data on sentiment analysis tools based on neural networks to draw conclusions about them. Nevertheless, according to the latest developments, this neural network approach delivers promising results [58,79].…”

Section: Future Research Directionsmentioning

confidence: 99%

“…In Section 5, we discuss our results, before concluding the paper in Section 6. described models based on the neural network BERT [14], which were trained with data related to SE such as GitHub 3 or Stack Overflow 4 . In their replication study, Novielli et al [58] explained some sentiment analysis tools (e.g. Senti4SD [8]) in great detail and described the underlying data.…”

Section: Introductionmentioning

confidence: 99%

Development and Application of Sentiment Analysis Tools in Software Engineering: A Systematic Literature Review

Obaidi

Klünder

2021

Evaluation and Assessment in Software Engineering

View full text Add to dashboard Cite

Software development is a collaborative task and, hence, involves different persons. Research has shown the relevance of social aspects in the development team for a successful and satisfying project closure. Especially the mood of a team has been proven to be of particular importance. Thus, project managers or project leaders want to be aware of situations in which negative mood is present to allow for interventions. So-called sentiment analysis tools offer a way to determine the mood based on text-based communication. In this paper, we present the results of a systematic literature review of sentiment analysis tools developed for or applied in the context of software engineering. Our results summarize insights from 80 papers with respect to (1) the application domain, (2) the purpose, (3) the used data sets, (4) the approaches for developing sentiment analysis tools and ( 5) the difficulties researchers face when applying sentiment analysis in the context of software projects. According to our results, sentiment analysis is frequently applied to open-source software projects, and most tools are based on support-vector machines. Despite the frequent use of sentiment analysis in software engineering, there are open issues, e.g., regarding the identification of irony or sarcasm, pointing to future research directions. CCS CONCEPTS• Software and its engineering → Collaboration in software development; • Human-centered computing → Collaborative and social computing systems and tools.

show abstract

Assessment of off-the-shelf SE-specific sentiment analysis tools: An extended replication study

Cited by 21 publications

References 43 publications

The Impacts of Sentiments and Tones in Community-Generated Issue Discussions

The Impacts of Sentiments and Tones in Community-Generated Issue Discussions

On the validity of pre-trained transformers for natural language processing in the software engineering domain

Development and Application of Sentiment Analysis Tools in Software Engineering: A Systematic Literature Review

Contact Info

Product

Resources

About