Embedding Java Classes with code2vec

Abstract

Automatic source code analysis in key areas of software engineering, such as code security, can benefit from Machine Learning (ML). However, many standard ML approaches require a numeric representation of data and cannot be applied directly to source code. Thus, to enable ML, we need to embed source code into numeric feature vectors while maintaining the semantics of the code as much as possible. code2vec is a recently released embedding approach that uses the proxy task of method name prediction to map Java methods to feature vectors. However, experimentation with code2vec shows that it learns to rely on variable names for prediction, causing it to be easily fooled by typos or adversarial attacks. Moreover, it is only able to embed individual Java methods and cannot embed an entire collection of methods such as those present in a typical Java class, making it difficult to perform predictions at the class level (e.g., for the identification of malicious Java classes). Both shortcomings are addressed in the research presented in this paper. We investigate the effect of obfuscating variable names during training of a code2vec model to force it to rely on the structure of the code rather than specific names and consider a simple approach to creating class-level embeddings by aggregating sets of method embeddings. Our results, obtained on a challenging new collection of source-code classification problems, indicate that obfuscating variable names produces an embedding model that is both impervious to variable naming and more accurately reflects code semantics. The datasets, models, and code are shared 1 for further ML research on source code. CCS CONCEPTS• Computing methodologies → Artificial intelligence; • Software and its engineering → Software creation and management; • Security and privacy → Intrusion/anomaly detection and malware mitigation; Software security engineering; Software reverse engineering.

An Exploratory Study on Code Attention in BERT

“…Therefore, semantic tasks should use upper-level layers to represent code to achieve the best performance. Interestingly, the performance remains uniform in the middle layers (4)(5)(6)(7)(8)(9)(10). This can be related to the uniform attention of identifiers, values we see in the Figure 4.…”

“…They compare the performance of these models from the perspective of probing classifiers. Other studies [5,9,35,49], show that identifiers are important code entities and can be used in the modeling of Transfoemr based models. This work presents the first study in software engineering, which analyses the multi-headed attention framework of BERT, which is not done previously [21].…”

A Study of Variable-Role-based Feature Enrichment in Neural Models of Code

“…While this work focused on extracting a set of handcrafted features for better transparency, we study how feature enrichment affects in model's training behavior. Recent studies have shown that state-of-the-art models heavily rely on variables [13,28], specific tokens [29], and even structures [30]. Chen et al [31] focus on semantic representations of program variables, and study how well models can learn similarity between variables that have similar meaning (e.g., minimum and minimal).…”

Assessing Robustness of ML-Based Program Analysis Tools using Metamorphic Program Transformations

“…Adversarial attacks. Related work stems from Compton et al [20] that introduces randomization of variable-names in the training dataset of a code2vec model for training data augmentation. Their study shows that the model trained on the augmented training dataset achieves slightly better accuracy than the model trained on the original dataset which motivates to systematically investigate for overfitting.…”