Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code

Abstract

Statistical language modeling techniques have successfully been applied to large source code corpora, yielding a variety of new software development tools, such as tools for code suggestion, improving readability, and API migration. A major issue with these techniques is that code introduces new vocabulary at a far higher rate than natural language, as new identifier names proliferate. Both large vocabularies and out-of-vocabulary issues severely affect Neural Language Models (NLMs) of source code, degrading their performance and rendering them unable to scale. In this paper, we address this issue by: 1) studying how various modelling choices impact the resulting vocabulary on a large-scale corpus of 13,362 projects; 2) presenting an open vocabulary source code NLM that can scale to such a corpus, 100 times larger than in previous work; and 3) showing that such models outperform the state of the art on three distinct code corpora (Java, C, Python). To our knowledge, these are the largest NLMs for code that have been reported. All datasets, code, and trained models used in this work are publicly available. CCS CONCEPTS• Software and its engineering → Software maintenance tools.

Learning to Extend Program Graphs to Work-in-Progress Code

“…While Bengio et al [11] reported worse results when using this estimator compared to straight-through, we found the latter to fail on our task. 4: Relation prediction model produces high quality edges on the validation and test splits of the Python corpus [37]. The model has an F score above 0.8 for all edge types, and attains an F score close to 1.0 for the most frequently occurring types.…”

“…We adopted the Python source code dataset used by Karampatsis and Sutton [36] that is comprised of two training splits (one small and one large), a validation, a test, and an encoding split used only for learning a tokenization. The corpus sanitized by Karampatsis et al [37] is licensed under CC BY 4.0. Dataset Preprocessing.…”

“…We also converted all files in Python2 to Python3 using Python's automated translation tool lib2to3. To reduce the size of the vocabulary, we followed Karampatsis et al [37] in splitting tokens into subtokens with the byte-pair encoding algorithm [64] ran for 10k merges using the Tokenizer package from Hugging Face. While the model presented in Section 3.3 is in principle able to handle tokens that consist of an arbitrary number of subwords, we truncate each token to have a maximum of 6 subwords for computational efficiency.…”

“…Since many files are longer than the typical context size of a Transformer-based model, which parts of a file should one train with becomes a question. Unlike Karampatsis et al [37], we did not take a fixed size prefix of each file. Instead, we sampled uniformly a new window each time a file is selected for a gradient update.…”

“…We do not compare directly against results in [37], since our setting is not strictly comparable to theirs due to subtle differences in data preprocessing (e.g. the specific procedure of creating subwords is different).…”