A Neural Model for Generating Natural Language Summaries of Program Subroutines

Abstract

Source code summarization -creating natural language descriptions of source code behavior -is a rapidly-growing research topic with applications to automatic documentation generation, program comprehension, and software maintenance. Traditional techniques relied on heuristics and templates built manually by human experts. Recently, data-driven approaches based on neural machine translation have largely overtaken template-based systems. But nearly all of these techniques rely almost entirely on programs having good internal documentation; without clear identifier names, the models fail to create good summaries. In this paper, we present a neural model that combines words from code with code structure from an AST. Unlike previous approaches, our model processes each data source as a separate input, which allows the model to learn code structure independent of the text in code. This process helps our approach provide coherent summaries in many cases even when zero internal documentation is provided. We evaluate our technique with a dataset we created from 2.1m Java methods. We find improvement over two baseline techniques from SE literature and one from NLP literature.

Automatically Extracting Subroutine Summary Descriptions from Unstructured Comments

“…Problem Statement The problem we target in this paper is the extraction of summary descriptions from unstructured subroutine comments. By "summary descriptions," we mean a short natural language explanation of code behavior or purpose (maximum 12 words, in line with related work [10], which found that most summary descriptions consisted of fewer than 13 word tokens). By "unstructured subroutine comments," we mean the long comments immediately preceding methods in source code.…”

“…Referring to the process as "summarization" alludes to a history of work in Natural Language Processing of extractive summarization of documents -early attempts at code summarization involved choosing a set of n important words from code [18], [19] and then converting those words into complete sentences by placing them into sentence templates [2], [20]- [22]. A 2016 survey [23] highlights these approaches around the time that a vast majority of code summarization techniques began to be based on neural networks trained from big data input [10], [14], [24]- [27]. These NN-based approaches have proliferated, but suffer an Achilles' heel of reliance on very large, clean datasets of examples of code comments.…”

“…[10], [24], [26], [27]) have explored neural-based representations of source code for the task of summarization. We integrate a recent representation described by LeClair et al [10] at ICSE'19 into the BiLSTM approach. Essentially, we augmented the encoder to accept the function source code as another input alongside the comment, but otherwise left the BiLSTM the same.…”

“…Therefore, a second strategy is to automatically generate the summaries based on patterns learned from big data input. This second strategy saves significant human effort, but relies on large numbers (on the order of millions [10]) of high-quality example summaries for learning. These examples are usually extracted from metadata within large code repositories, but suitable metadata is scarce.…”

“…There does exist a large, untapped resource of summary descriptions in the form of unstructured header comments found in source code. Unstructured comments are much more numerous than the well-structured ones in metadata (over 3x as many in one dataset [10]), but are much longer and more expansive in scope than short summary descriptions. As we show later in this paper, these comments nearly always have a short summary description embedded in them, but the summary may occur in many locations: surrounded by different text, commented-out code, or even diagrams or logos as ASCII art.…”