On the naturalness of software

Abstract

Natural languages like English are rich, complex, and powerful. The highly creative and graceful use of languages like English and Tamil, by masters like Shakespeare and Avvaiyar, can certainly delight and inspire. But in practice, given cognitive constraints and the exigencies of daily life, most human utterances are far simpler and much more repetitive and predictable. In fact, these utterances can be very usefully modeled using modern statistical methods. This fact has led to the phenomenal success of statistical approaches to speech recognition, natural language translation, question-answering, and text mining and comprehension. We begin with the conjecture that most software is also natural , in the sense that it is created by humans at work, with all the attendant constraints and limitations---and thus, like natural language, it is also likely to be repetitive and predictable. We then proceed to ask whether (a) code can be usefully modeled by statistical language models and (b) such models can be leveraged to support software engineers. Using the widely adopted n -gram model, we provide empirical evidence supportive of a positive answer to both these questions. We show that code is also very regular, and, in fact, even more so than natural languages. As an example use of the model, we have developed a simple code completion engine for Java that, despite its simplicity, already improves Eclipse's completion capability. We conclude the paper by laying out a vision for future research in this area.

CodeGRU: Context-aware deep learning with gated recurrent unit for source code modeling

“…In this work, we consider a large size of vocabulary for modeling source code without compromising any source code token. Unlike previous works [49,25], we do not remove any source code tokens which helps CodeGRU to capture source code regularities much more effectively. To build the vocabulary first, we tokenize the source code files at token level as shown in Fig.…”

“…To answer the research question (RQ1), we compare the performance of the proposed approach with the state-of-the-art approaches [49,25,34] in order to find out the performance improvement of the proposed approach. To answer the research question (RQ2), we evaluated the CodeGRU with mean reciprocal rank(MRR) with state-of-art approaches [49,25,34] in order to evaluate how well CodeGRU performs in term of source code suggestion and generation tasks.…”

“…To build our code database, we collected open-source java projects from GitHub summarized 1 in Table 4. We choose the projects used in previous references [49,25,34]. The Table 4 shows the version of the projects, files count in each project, total number of code lines, total code tokens and unique code tokens found in each project without any transformation.…”

“…RNN+ is a variation of vanilla RNN trained by using our proposed approach. GRU based model is trained similer to previous works [25,49] by treating source code as simple text and with fixed size window approach. The CodeGRU model is trained by using our proposed approach with variable size context as discribed earlier in section 4.…”

“…The advancement in the deep neural network (DNN) based NLP models [52] have recently shown that they can effectively overcome the context issue that cannot be effectively addressed by SLM [41,25,46] based models. Many deep learning based approaches have been applied for different tasks for source code modeling such as code summarization [26,4], code generation [43], error fixing [43,22], and code recommendation [49,20,14].…”