Comparison of Image-Based and Text-Based Source Code Classification Using Deep Learning

Abstract

Source code classification (SCC) is a task to assign codes into different categories according to a criterion such as according to their functionalities, programming languages or vulnerabilities. Many source code archives are organized according to the programming languages, and thereby, the desired code fragments can be easily accessed by searching within the archive. However, manually organizing source code archives by field experts is labor intensive and impractical because of the fastgrowing available source codes. Therefore, this study proposes new convolutional neural network (CNN) architectures to build source code classifiers that automatically identify programming languages from source codes. This is the first study in which the performances of deep learning algorithms on programming language identification are compared on both image and text files. In this study, the experiments are performed on three source code datasets to identify eight programming languages, including C, C++, C# , Go, Python, Ruby, Rust, and Java. The comparative results indicate that although textbased SCC and image-based SCC approaches achieve very high ( > 93.5% ) and similar accuracies, text-based classification has significantly better performance in terms of execution time.

Image-based many-language programming language identification

“…With respect to starting from scratch, our approach offers the benefits of cheaper (re)training, reducing maintenance costs. Considering the very marginal reduction in precision in comparison to previous work (≤ 1.5% with respect to Kiyak et al (2020)), which is probably in large part imputable to the much higher language diversity in our experiments, the pros/cons balance seems to tilt towards pretrained CNNs and transfer learning. In this respect it seems worth to explore side-tuning (Zhang et al, 2020), a recent technique for transfer learning which consists in adapting a pre-trained network by training a lightweight ''side'' network that is then fused with the (unchanged) pre-trained network via summation.…”

“…Image-based PLI Kiyak et al (2020) compared several image-and text-based approaches to Programming Language Identification (PLI). At a glance, Table 1 in their work reports that the maximum diversity supported by image-based PLI among surveyed works was 8 languages, reached by the same authors in Kiyak et al (2020) with an accuracy of 93.5% on a dataset of 40 K files. We achieve comparable performances (92% precision and recall) with much higher language diversity (149 languages) and on a larger dataset (300 K snippets).…”

“…They use real images from screencasts and labelling performed manually by students, whereas we use synthetic images and rely on Linguist (GitHub, Inc., 2011) as source of truth. Hong, Mizuno & Kondo (2019) (not considered in Kiyak et al (2020)) performed imagebased PLI over 10 languages with 90% accuracy, using snippets from StackOverflow and GitHub, rendering them to bitmaps like we do, but using GuessLang (SOMDA, 2017) as source of truth. They use ResNet as a pretrained CNN, which we also considered in this work.…”

“…Depending on the use case, two major classes of ML approaches to PLI have been used ( Kiyak et al, 2020 ): text-based and image-based programming language identification. In text-based approaches source code is represented as character sequences, such as files stored in version control system (VCS) repositories.…”

“…Image-based PLI models are currently capable of recognizing a limited amount of different programming languages, with a maximum “diversity” of 10 languages found in the literature ( Kiyak et al, 2020 ; Hong, Mizuno & Kondo, 2019 ). Arguably, recognizing a handful of programming languages could be approached with simple heuristics without incurring the maintenance overhead of (re-)training machine learning models.…”