Do Programmers Prefer Predictable Expressions in Code?

Abstract

Source code is a form of human communication, albeit one where the information shared between the programmers reading and writing the code is constrained by the requirement that the code executes correctly. Programming languages are more syntactically constrained than natural languages, but they are also very expressive, allowing a great many different ways to express even very simple computations. Still, code written by developers is highly predictable, and many programming tools have taken advantage of this phenomenon, relying on language model surprisal as a guiding mechanism. While surprisal has been validated as a measure of cognitive load in natural language, its relation to human cognitive processes in code is still poorly understood. In this paper, we explore the relationship between surprisal and programmer preference at a small granularity—do programmers prefer more predictable expressions in code? Using meaning‐preserving transformations, we produce equivalent alternatives to developer‐written code expressions and run a corpus study on Java and Python projects. In general, language models rate the code expressions developers choose to write as more predictable than these transformed alternatives. Then, we perform two human subject studies asking participants to choose between two equivalent snippets of Java code with different surprisal scores (one original and transformed). We find that programmers do prefer more predictable variants, and that stronger language models like the transformer align more often and more consistently with these preferences.

CONCORD: Clone-Aware Contrastive Learning for Source Code

“…During pre-training, Corder takes AST-based intermediate representation as input and ignores learning the source code text directly. Such ignorance could make the model less capable of understanding the rich semantics underneath the source code text, such as variable/function names and comments, which are the main resources to expose developers' intentions during coding [8][9][10], and consequently, degrade the quality of learned code representations. ContraCode.…”

“…While the optimized and obfuscated code provides precise and formal semantics [8], they tend to be unnatural, introducing data structures and variable names that are not commonly used in human-written programs. Existing studies have argued that such formal but unnatural programs are less favorable to human developers [9,10] and obstruct the code models' learning [11]. Also, ContraCode does not generate semantically contradicting programs as hard negative samples.…”

NatGen: Generative pre-training by "Naturalizing" source code

“…The formal channel, unique to code, affords precise, formal semantics; interpreters, compilers, etc., use this channel. On the other hand, the natural channel (perhaps more probabilistic and noisy) relies on variable names, comments, etc., and is commonly used by humans for code comprehension and communication [15,16]. The formal channel's precision enables semantic preserving code transformation, which supports static analysis, optimization, obfuscation, etc.…”

“…However, not all the semantically equivalent code is "natural" [33]-the usual way developers write code and thus, amenable to statistical models [33]. In fact, deviation from such "naturalness" may lead to unintended bugs [53], and increase difficulty of human comprehension [15,16].…”

“…Studies, with human-subjects[15,16] suggest that humans find such rewritten but semantically identical forms harder to read and understand.…”