Toward mining "concept keywords" from identifiers in large software projects

Abstract

We propose the Concept Keyword Term Frequency/Inverse Document Frequency (ckTF/IDF) method as a novel technique to efficiency mine concept keywords from identifiers in large software projects. ckTF/IDF is suitable for mining concept keywords, since the ckTF/IDF is more lightweight than the TF/IDF method, and the ckTF/IDF's heuristics is tuned for identifiers in programs.We then experimentally apply the ckTF/IDF to our educational operating system udos, consisting of around 5,000 lines in C code, which produced promising results; the udos's source code was processed in 1.4 seconds with an accuracy of around 57%. This preliminary result suggests that our approach is useful for mining concept keywords from identifiers, although we need more research and experience.

Learning natural coding conventions

“…High quality identifier names lie at the heart of software engineering [6,16,20,24,42,66,69,54]; they drive code readability and comprehension [12,19,20,41,44,68]. According to Deißenböck and Pizka [17], identifiers represent the majority (70%) of source code tokens.…”

Mining Change Logs and Release Notes to Understand Software Maintenance and Evolution

“…• Observation 2: The information contained in the text of change logs and release notes of a software product is represented with some keywords. In the two types of observations, Observation 2 has been widely accepted in the text mining community, i.e., keyword mining has become a standard text mining technique [10]; Observation 1 is supported by the evidences reported by previous studies. For example, in [6], Baysal and Malton found that the non-source code documents contain similar amount of contents of source code changes in software maintenance and evolution, which indicates that non-source code documents, such as email archives, release notes, and change logs, might accurately record the maintenance and evolution activity of a software product.…”

Natural Language-Based Software Analyses and Tools for Software Maintenance

“…Contextual query reformulation relies on SWUM's phrasal concepts to extract phrases from source code because existing techniques for extracting phrases did not meet the needs of the concern location problem. There is work on automatically extracting topic words and phrases from source code [67,71], displaying search results in a concept lattice of keywords [72], and clustering program elements that share similar phrases [46]. Although useful for exploring the overall word usage of an unfamiliar software system, these techniques are not sufficient for exploring all usage.…”

“…Although useful for exploring the overall word usage of an unfamiliar software system, these techniques are not sufficient for exploring all usage. In contrast to the contextual approach, these approaches either filter the topics based on perceived importance to the system [46,71,72], or do not produce human understandable topic labels [67]. Since it is impossible to predict a priori what will be of interest to the developer, the contextual approach lets the developer filter the results with a natural language query, and uses human-readable extracted phrases.…”