On the naturalness of software

Abstract

Natural languages like English are rich, complex, and powerful. The highly creative and graceful use of languages like English and Tamil, by masters like Shakespeare and Avvaiyar, can certainly delight and inspire. But in practice, given cognitive constraints and the exigencies of daily life, most human utterances are far simpler and much more repetitive and predictable. In fact, these utterances can be very usefully modeled using modern statistical methods. This fact has led to the phenomenal success of statistical approaches to speech recognition, natural language translation, question-answering, and text mining and comprehension.We begin with the conjecture that most software is also natural, in the sense that it is created by humans at work, with all the attendant constraints and limitations-and thus, like natural language, it is also likely to be repetitive and predictable. We then proceed to ask whether a) code can be usefully modeled by statistical language models and b) such models can be leveraged to support software engineers. Using the widely adopted n-gram model, we provide empirical evidence supportive of a positive answer to both these questions. We show that code is also very repetitive, and in fact even more so than natural languages. As an example use of the model, we have developed a simple code completion engine for Java that, despite its simplicity, already improves Eclipse's built-in completion capability. We conclude the paper by laying out a vision for future research in this area.

Mining source code repositories at massive scale using language modeling

“…In those areas, they are used for ranking candidate sentences, such as candidate translations of a foreign language sentence, based on how natural they are in the target language. To our knowledge, Hindle et al [1] were the first to apply language models to source code.…”

“…Recently, Hindle et al [1] presented pioneering work in learning language models over source code, that represent broad statistical characteristics of coding style. Language models (LMs) are simply probability distributions over strings.…”

SourcererCC

“…While there are many ways in which tokens in a block can be ordered e.g., alphabetical order, length of tokens, occurance frequency of token in a corpus, etc., a natural question is what order is most effective in this context. As it turns out, software vocabulary exhibits very similar characteristics to natural languages corpus and also follow Zipf's law [13,40]. That is, there are few very popular (frequent) tokens, and the frequency of tokens decreases very rapidly with rank.…”

Finding and correcting syntax errors using recurrent neural networks

“…The mental model of the programmer may be something like a language model for speech, but rather applied to code. Language models are typically applied to natural human utterances but they have also been successfully applied to software (Hindle et al, 2012;Raychev et al, 2014;White et al, 2015), and can be used to discover unexpected segments of tokens in source code (Campbell et al, 2014).…”

“…Thus GrammarGuru uses language models to capture code regularity or naturalness and then looks for irregular code (Campbell et al, 2014). Once the location of a potential error is found, code completion techniques that exploit language models (Hindle et al, 2012;Raychev et al, 2014;White et al, 2015) can be used to suggest possible fixes. Traditional parsers do not rely upon such information.…”