Automatically identifying performance issue reports with heuristic linguistic patterns

Zhao, Yulan; Xiao, Lu; Babvey, Pouria; Lei, Sun; Wong, Sunny; Martı́nez, Amalia; Xiao, Wang

doi:10.1145/3368089.3409674

Cited by 5 publications

(18 citation statements)

References 62 publications

(47 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When developers explain something similar, they are likely to use recurrent textual expressions in their comments [63]. Existing studies frequently rely on such expressions to extract linguistic patterns in order to facilitate context understanding and task automation [17,40,49,58,69]. In this section, we present a novel approach, iLead, which Find Key Sentences Fig.…”

Section: Approachmentioning

confidence: 99%

“…Specifically, for each new pattern, the algorithm will iteratively examine the impact of merging it into the target pattern ranking list, and only accept it when it positively contributes to the prediction performance of iLead. Following an existing study [40,58,69], we choose the F1-Score (mentioned in 5.4) as the performance metric. Specifically, the algorithm will probe every candidate position in the pattern ranking list for inserting a new pattern, calculate the performance of iLead in each probing, record the best performance and the corresponding optimal insertion position.…”

Section: 31mentioning

confidence: 99%

“…In addition, we further explore whether our extracted patterns are helpful to the machine/deep learning models, following the existing studies [40,58,69]. In detail, we enrich the model inputs with the knowledge about pattern matching, which will be consider as a boolean vector, i.e., if a comment match with certain patterns, the corresponding position of the vector is set to 1, otherwise it is 0.…”

Section: Experiments For Answering Rq2mentioning

confidence: 99%

“…Previous researches noticed that people tend to use recurrent expressions to illustrate similar things, and leverage such phenomenon, existing researches constructed the linguistic patterns for automating various software engineering tasks. Zhao et al [69] conducted a manual analysis of 980 performance issue reports and extracted 80 linguistic patterns; they then combined these patterns with machine/deep learning algorithms to automatically detect the performance issue reports. Lin et al [40] manually analyzed 388 API-related sentences and defined 157 linguistic patterns, and used these patterns to classify the opinions about API on Q&A website.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

Identifying Emergent Leadership in OSS Projects Based on Communication Styles

Huang¹,

Yang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

In open source software (OSS) communities, existing leadership indicators are dominantly measured by code contribution or community influence. Recent studies on emergent leadership shed light on additional dimensions such as intellectual stimulation in collaborative communications. To that end, this paper proposes an automated approach, named iLead, to mine communication styles and identify emergent leadership behaviors in OSS communities, using issue comments data. We start with the construction of 6 categories of leadership behaviors based on existing leadership studies. Then, we manually label leadership behaviors in 10,000 issue comments from 10 OSS projects, and extract 304 heuristic linguistic patterns which represent different types of emergent leadership behaviors in flexible and concise manners. Next, an automated algorithm is developed to merge and consolidate different pattern sets extracted from multiple projects into a final pattern ranking list, which can be applied for the automatic leadership identification. The evaluation results show that iLead can achieve a median precision of 0.82 and recall of 0.78, outperforming ten machine/deep learning baselines.To demonstrate practical usefulness, we also conduct empirical analysis and human evaluation of the identified leadership behaviors from iLead. We argue that emergent leadership behaviors in issue discussion should be taken into consideration to broaden existing OSS leadership viewpoints. Practical insights on community building and leadership skill development are offered for OSS community and individual developers, respectively. CCS Concepts: • Software and its engineering → Open source model.

show abstract

Section: Approachmentioning

confidence: 99%

Section: 31mentioning

confidence: 99%

Section: Experiments For Answering Rq2mentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Identifying Emergent Leadership in OSS Projects Based on Communication Styles

Huang¹,

Yang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…By manually learning from 980 real-life performance issue reports tagged by developers in Apache's JIRA issue tracking system, we derived a set of 80 HLPs that recur in these issue reports and capture common linguistic features of how performance issues are described. Admittedly, this was a labor-intensive process of approximately 762 human hours, cross-validated among four taggers to ensure reliability [12]. However, this manual effort is an up-front, one-time investment.…”

Section: Introductionmentioning

confidence: 99%

A Platform-Agnostic Framework for Automatically Identifying Performance Issue Reports With Heuristic Linguistic Patterns

Zhao,

Xiao,

Wong

2024

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Software performance is critical for system efficiency, with performance issues potentially resulting in budget overruns, project delays, and market losses. Such problems are reported to developers through issue tracking systems, which are often under-tagged, as the manual tagging process is voluntary and time-consuming. Existing automated performance issue tagging techniques, such as keyword matching and machine/deep learning models, struggle due to imbalanced datasets and a high degree of variance. This paper presents a novel hybrid classification approach, combining Heuristic Linguistic Patterns (HLPs) with machine/deep learning models to enable practitioners to automatically identify performance-related issues. The proposed approach works across three progressive levels: HLP tagging, sentence tagging, and issue tagging, with a focus on linguistic analysis of issue descriptions. The authors evaluate the approach on three different datasets collected from different projects and issue-tracking platforms to prove that the proposed framework is accurate, project-and platform-agnostic, and robust to imbalanced datasets. Furthermore, this study also examined how the two unique techniques of the framework, including the fuzzy HLP matching and the Issue HLP Matrix, contribute to the accuracy. Finally, the study explored the effectiveness and impact of two off-the-shelf feature selection techniques, Boruta and RFE, with the proposed framework. The results showed that the proposed framework has great potential for practitioners to accurately (with up to 100% precision, 66% recall, and 79% F1-score) identify performance issues, with robustness to imbalanced data and good transferability to new projects and issue tracking platforms.

show abstract