Recommending Comprehensive Solutions for Programming Tasks by Mining Crowd Knowledge

Abstract

Developers often search for relevant code examples on the web for their programming tasks. Unfortunately, they face two major problems. First, the search is impaired due to a lexical gap between their query (task description) and the information associated with the solution. Second, the retrieved solution may not be comprehensive, i.e., the code segment might miss a succinct explanation. These problems make the developers browse dozens of documents in order to synthesize an appropriate solution. To address these two problems, we propose CROKAGE (Crowd Knowledge Answer Generator), a tool that takes the description of a programming task (the query) and provides a comprehensive solution for the task. Our solutions contain not only relevant code examples but also their succinct explanations. Our proposed approach expands the task description with relevant API classes from Stack Overflow Q&A threads and then mitigates the lexical gap problems. Furthermore, we perform natural language processing on the top quality answers and then return such programming solutions containing code examples and code explanations unlike earlier studies. We evaluate our approach using 97 programming queries, of which 50% was used for training and 50% was used for testing, and show that it outperforms six baselines including the state-of-art by a statistically significant margin. Furthermore, our evaluation with 29 developers using 24 tasks (queries) confirms the superiority of CROKAGE over the state-of-art tool in terms of relevance of the suggested code examples, benefit of the code explanations and the overall solution quality (code + explanation).

Extracting Semantics from Question-Answering Services for Snippet Reuse

“…The second experiment relies on post links and is used to provide proof that our system is effective (and minimize possible threats to validity). Finally, for our third experiment, we compare StackSearch to the tool CROKAGE [24], which is quite similar to our system. Comparing StackSearch with other approaches was not possible, since several systems are not maintained and/or they are not publicly available (to facilitate researchers with similar challenges, we uploaded our code at https://github.com/AuthEceSoftEng/StackSearch).…”

“…This system, however, is also largely based on synthesizing API calls and does not focus on semantic retrieval. Finally, an even more recent system is CROKAGE [24], which employs embeddings and further expands the query with relevant API classes from Stack Overflow. The final results are ranked according to multiple factors, including their lexical and semantic similarity with the query and their similar API usage.…”

Essential Sentences for Navigating Stack Overflow Answers

“…Soliman et al's [15] approach on the other hand is more domain-specific, focusing on how architects search for architecturally relevant information on Stack Overflow. CROKAGE by Silva et al [16] takes the description of a programming task and provides a comprehensive solution for this task by searching multiple threads. In contrast, our work focuses on the navigation of a single thread, with the goal of identifying navigational cues.…”

“…Recall that participants answer this question at the end of the survey, allowing them to reflect on all sentences they evaluated. The set of codes we use, along with the number of instances per code in parentheses is as follows: direct solution (16), explanation (13), relevance of info (11), code/lib (6), wellwritten (5), SO info (2), alternate solution (1), and warning (1). Our inter-rater agreement kappa score is 0.86.…”

“…RQ 3 Are essential sentences identified by different approaches perceived as helpful navigational cues? RQ 4 What information is available in essential sentences?…”