When deep learning met code search

Abstract

There have been multiple recent proposals on using deep neural networks for code search using natural language. Common across these proposals is the idea of embedding code and natural language queries, into real vectors and then using vector distance to approximate semantic correlation between code and the query. Multiple approaches exist for learning these embeddings [15,19,24,26], including unsupervised techniques, which rely only on a corpus of code examples, and supervised techniques, which use an aligned corpus of paired code and natural language descriptions. The goal of this supervision is to produce embeddings that are more similar for a query and the corresponding desired code snippet.Clearly, there are choices in whether to use supervised techniques at all, and if one does, what sort of network and training to use for supervision. This paper is the first to evaluate these choices systematically. To this end, we assembled implementations of stateof-the-art techniques to run on a common platform, training and evaluation corpora. To explore the design space in network complexity, we also introduced a new design point that is a minimal supervision extension to an existing unsupervised technique.Our evaluation shows that: 1. adding supervision to an existing unsupervised technique can improve performance, though not necessarily by much; 2. simple networks for supervision can be more effective that more sophisticated sequence-based networks for code search; 3. while it is common to use docstrings to carry out supervision, there is a sizeable gap between the effectiveness of docstrings and a more query-appropriate supervision corpus.

FACER: An API Usage-based Code-example Recommender for Opportunistic Reuse

“…Several of the latest code search techniques that find code given a natural language query rely on machine learning techniques (e.g.,NCS [10], DeepCS [8], UNIF [38], MMAN [39], TBCAA [40], and CoaCor [41]). NCS proposes an enhanced word embedding for a natural language query [10].…”

“…This unified representation bridges the lexical gap between queries and source code resulting in relevant code fragments that do not necessarily contain query words. UNIF [38] is an extension of NCS that adds supervision to modify embeddings during training with the overall effect of improving the performance for code search. MMAN [39] is a Multi-Modal Attention Network for semantic source code retrieval.…”

AMS: generating AutoML search spaces from weak specifications

“…We chose to sample 50 classes as this covers approximately 28% of the components available in Scikit-Learn and balanced the need for detailed manual annotation. For each query, we retrieved the top 10 API components based on: 1) our BM25 metric, 2) cosine similarity using averaged pre-trained neural embeddings (which have been shown to be effective for the related task of code search [6]), and 3) a uniform random metric. We used (2) to compare the use of BM25 with another unsupervised approach to semantic similarity.…”

“…To compare AL and AMS, we consider the weak specification of Scikit-Learn components 6 : { L o g i s t i c R e g r e s s i o n , LinearSVC , S t a n d a r d S c a l e r } 6 names abbreviated for brevity and run experiments on our 9 datasets. We use 5-fold CV, pair pipelines between CV folds in order to appropriately perform comparisons after removing pipelines that don't satisfy the weak specification, and then compute wins on the paired pipelines.…”

“…At a smaller scale, automated example extraction [24] from specific scripts allows allows users to produce minimal working snippets. Large-scale corpora that exercise particular APIs can be used to mine preconditions for method calls [33], order-based specifications for chaining calls [2], code repair patterns [31], and drive semantic code search [6,25,30].…”