Neural Code Search Evaluation Dataset

Li, Hongyu; Kim, Seohyun; Chandra, Satish

doi:10.48550/arxiv.1908.09804

Cited by 12 publications

(17 citation statements)

References 4 publications

(11 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, Aroma was proved to be effective in identifying similarities between partial code snippets, e.g., obtained from STACKOVERFLOW. Similar to other contributions [32], [33], we use Aroma to define a metric for the similarity between the answers in our evaluation set. This metric is intended to mimic the manual assessment of the correctness of search results but in an automatic and reproducible way [33], without relying on the human judgment that, considering the size of our dataset, would be infeasible.…”

Section: Mean Reciprocal Rank (Mrr)mentioning

confidence: 99%

On the Effectiveness of Transfer Learning for Code Search

Salza,

Schwizer,

et al. 2021

Preprint

View full text Add to dashboard Cite

The Transformer architecture and transfer learning have marked a quantum leap in natural language processing, improving the state of the art across a range of text-based tasks. This paper examines how these advancements can be applied to and improve code search. To this end, we pre-train a BERT-based model on combinations of natural language and source code data and evaluate it on pairs of StackOverflow question titles and code answers. Our results show that the pre-trained models consistently outperform the models that were not pre-trained. In cases where the model was pre-trained on natural language "and" source code data, it also outperforms an information retrieval baseline based on Lucene. Also, we demonstrated that combined use of an information retrieval-based approach followed by a Transformer, leads to the best results overall, especially when searching into a large search pool. Furthermore, transfer learning is particularly effective when much pre-training data is available and fine-tuning data is limited. We demonstrate that natural language processing models based on the Transformer architecture can be directly applied to source code analysis tasks, such as code search. With the development of Transformer models designed more specifically for dealing with source code data, we believe the results on source code analysis tasks can be further improved.

show abstract

Section: Mean Reciprocal Rank (Mrr)mentioning

confidence: 99%

On the Effectiveness of Transfer Learning for Code Search

Salza,

Schwizer,

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, an interesting direction of software engineering is to use machine/deep learning for different tasks to improve software development. Such as code search (e.g., [2,24,31,39]), clone detection (e.g., [7,18,19,64,67]), program repair (e.g,. [10,45,60,66]), document (such as API and questions/answers/tags) recommendation (e.g., [22,25,26,55,63,65,69,70,76]).…”

Section: Machine/deep Learning On Software Engineeringmentioning

confidence: 99%

“…All the code snippets are embedded into a high-dimensional vector space by our approach. A variety of applications such as code search (e.g., [24,31,39]) , summarization (e.g., [30,32,33,62]), retrieval (e.g., [1,9,71]), and API recommendation (e.g., [25,26]) can benefit from the code embeddings used in our study.…”

Section: The Problem and Our Solutionmentioning

confidence: 99%

“…Recently, deep learning has achieved promising results in solving many software engineering tasks, such as code search (e.g., [24,31,39]), code summarization (e.g., [30,32,33,62]), and API recommendation (e.g., [25,26]). Among these works, a number of researchers have applied the sequence to sequence methods for mining the ⟨natural language, code snippet⟩ pairs, such as the commit message generation.…”

Section: Deep Sequence To Sequence Approachmentioning

confidence: 99%

See 1 more Smart Citation

Generating Question Titles for Stack Overflow from Mined Code Snippets

Gao,

Xia,

Grundy

et al. 2020

Preprint

View full text Add to dashboard Cite

Stack Overflow has been heavily used by software developers as a popular way to seek programming-related information from peers via the internet. The Stack Overflow community recommends users to provide the related code snippet when they are creating a question to help others better understand it and offer their help. Previous studies have shown that a significant number of these questions are of low-quality and not attractive to other potential experts in Stack Overflow. These poorly asked questions are less likely to receive useful answers and hinder the overall knowledge generation and sharing process. Considering one of the reasons for introducing low-quality questions in SO is that many developers may not be able to clarify and summarize the key problems behind their presented code snippets due to their lack of knowledge and terminology related to the problem, and/or their poor writing skills, in this study we propose an approach to assist developers in writing high-quality questions by automatically generating question titles for a code snippet using a deep sequence-to-sequence learning approach. Our approach is fully data-driven and uses an attention mechanism to perform better content selection, a copy mechanism to handle the rare-words problem and a coverage mechanism to eliminate word repetition problem. We evaluate our approach on Stack Overflow datasets over a variety of programming languages (e.g., Python, Java, Javascript, C# and SQL) and our experimental results show that our approach significantly outperforms several state-of-the-art baselines in both automatic and human evaluation. We have released our code and datasets to facilitate other researchers to verify their ideas and inspire the follow up work.

show abstract

“…In summary, our contributions to the field of code-to-code recommendation in this paper are four-fold: [17] and the Neural Code Search evaluation dataset [28] and find that the snippet lengths are heavily skewed, following a power-law distribution, with the vast majority of the snippets being short in length, and a long tail of longer snippets. We argue that code-to-code recommendation engines, to return concise and useful snippets, should implement techniques to counteract the bias caused by this skewness.…”

Section: Introductionmentioning

confidence: 99%

Senatus -- A Fast and Accurate Code-to-Code Recommendation Engine

Silavong,

Moran,

Georgiadis

et al. 2021

Preprint

View full text Add to dashboard Cite

Machine learning on source code (MLOnCode) is a popular research field that has been driven by the availability of large-scale code repositories and the development of powerful probabilistic and deep learning models for mining source code. Code-to-code recommendation is a task in MLOnCode that aims to recommend relevant, diverse and concise code snippets that usefully extend the code currently being written by a developer in their development environment (IDE). Code-to-code recommendation engines hold the promise of increasing developer productivity by reducing context switching from the IDE and increasing code-reuse. Existing code-to-code recommendation engines do not scale gracefully to large codebases, exhibiting a linear growth in query time as the code repository increases in size. In addition, existing code-to-code recommendation engines fail to account for the global statistics of code repositories in the ranking function, such as the distribution of code snippet lengths, leading to sub-optimal retrieval results. We address both of these weaknesses with Senatus, a new code-to-code recommendation engine. At the core of Senatus is De-Skew LSH a new locality sensitive hashing (LSH) algorithm that indexes the data for fast (sub-linear time) retrieval while also counteracting the skewness in the snippet length distribution using novel abstract syntax tree-based feature scoring and selection algorithms. We evaluate Senatus via automatic evaluation and with an expert developer user study and find the recommendations to be of higher quality than competing baselines, while achieving faster search. For example, on the CodeSearchNet dataset we show that Senatus improves performance by 6.7% F1 and query time 16x is faster compared to Facebook Aroma on the task of code-to-code recommendation.

show abstract

Neural Code Search Evaluation Dataset

Cited by 12 publications

References 4 publications

On the Effectiveness of Transfer Learning for Code Search

On the Effectiveness of Transfer Learning for Code Search

Generating Question Titles for Stack Overflow from Mined Code Snippets

Senatus -- A Fast and Accurate Code-to-Code Recommendation Engine

Contact Info

Product

Resources

About