Thinking Like a Developer? Comparing the Attention of Humans with Neural Models of Code

Paltenghi, Matteo; Pradel, Michael

doi:10.1109/ase51524.2021.9678712

Cited by 20 publications

(14 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides our work, there have been other studies that also try to explain the mechanisms of pre-trained models for code [1,30,32,38]. Karmakar and Robbes [24] applied four probing tasks on pretrained code models to investigate whether pre-trained models can learn different aspects of source code such as syntactic, structural, surface-level, and semantic information.…”

Section: Related Work 71 Understanding Pre-trained Models For Codementioning

confidence: 99%

Diet Code is Healthy: Simplifying Programs for Pre-Trained Models of Code

Zhang,

Shen

et al. 2022

Preprint

View full text Add to dashboard Cite

Pre-trained code representation models such as CodeBERT have demonstrated superior performance in a variety of software engineering tasks, yet they are often heavy in complexity, quadratically with the length of input sequence. Our empirical analysis on CodeBERT's attention reveals that CodeBERT pays more attention to certain types of tokens and statements such as keywords and data-relevant statements. Based on these findings, we propose Diet-CodeBERT, which aims at lightweight leverage of large pre-trained models for source code. DietCodeBERT simplifies the input program of CodeBERT with three strategies, namely, word dropout, frequency filtering, and an attention-based strategy which selects statements and tokens that receive the most attention weights during pre-training. Hence, it gives a substantial reduce in the computational cost without hampering the model performance. Results on two downstream tasks show that DietCodeBERT provides comparable results as CodeBERT with 40% less computational cost in fine-tuning and testing. CCS CONCEPTS• Computing methodologies → Natural language processing.

show abstract

Section: Related Work 71 Understanding Pre-trained Models For Codementioning

confidence: 99%

Diet Code is Healthy: Simplifying Programs for Pre-Trained Models of Code

Zhang,

Shen

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Attention studies of neural models of code. Paltenghi & Pradel (2021) have compared the attention weights of neural models of code and developers' visual attention when performing a code summarization task, and found a strong positive correlation on the copy attention mechanism for an instance of a pointer network (Vinyals et al, 2015). Wan et al (2022) and have then shown how the attention weights of pre-trained models on source code capture important properties of the abstract syntax tree of the program.…”

Section: Relation To Existing Workmentioning

confidence: 99%

“…We study four approaches: max, mean, rollout and follow-up attention. Apart from the rollout attention, which has been introduced by Abnar & Zuidema (2020), the other three are either inspired by the work of Paltenghi & Pradel (2021) or a novel contribution of this work, such as the follow-up attention.…”

Section: Extraction Functions For Interaction Matrixmentioning

confidence: 99%

“…These models are often based on the attention mechanism (Bahdanau et al, 2016), a key component of the transformer architecture (Vaswani et al, 2017). Besides providing substantial performance benefit, attention weights have been used to provide interpretability of neural models (Lin et al, 2017;Vashishth et al, 2019;Paltenghi & Pradel, 2021). In particular, Wan et al (2022) and Vig & Belinkov (2019) have shown how the attention weights contain important syntactic information on both the Abstract Syntax Tree (AST) of source code and Part of Speech (POS) tags in natural language.…”

Section: Introductionmentioning

confidence: 99%

“…There are datasets tracking developers' visual attention while looking at code, but they do not seem suitable to this task. The largest ones either put the developers in an unnatural (and thus possibly biasing) environment where most of the vision is blurred (Paltenghi & Pradel, 2021), or they contain few and very specific code comprehension tasks (Bednarik et al, 2020) on code snippets too short to exhibit any interesting code navigation pattern. This work.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Extracting Meaningful Attention on Source Code: An Empirical Study of Developer and Neural Model Code Exploration

Paltenghi¹,

Pandita²,

Henley³

et al. 2022

Preprint

View full text Add to dashboard Cite

The high effectiveness of neural models of code, such as OpenAI Codex and Al-phaCode, suggests coding capabilities of models that are at least comparable to those of humans. However, previous work has only used these models for their raw completion, ignoring how the model reasoning, in the form of attention weights, can be used for other downstream tasks. Disregarding the attention weights means discarding a considerable portion of what those models compute when queried. To profit more from the knowledge embedded in these large pre-trained models, this work compares multiple approaches to post-process these valuable attention weights for supporting code exploration. Specifically, we compare to which extent the transformed attention signal of CodeGen, a large and publicly available pretrained neural model, agrees with how developers look at and explore code when each answering the same sense-making questions about code. At the core of our experimental evaluation, we collect, manually annotate, and open-source a novel eye-tracking dataset comprising 25 developers answering sense-making questions on code over 92 sessions. We empirically evaluate five attention-agnostic heuristics and ten attention-based post processing approaches of the attention signal against our ground truth of developers exploring code, including the novel concept of follow-up attention which exhibits the highest agreement. Beyond the dataset contribution and the empirical study, we also introduce a novel practical application of the attention signal of pre-trained models with completely analytical solutions, going beyond how neural models' attention mechanisms have traditionally been used.

show abstract

Guidelines for using financial incentives in software-engineering experimentation

Krüger,

Çalıklı,

Bershadskyy

et al. 2024

Empir Software Eng

View full text Add to dashboard Cite

Context: Empirical studies with human participants (e.g., controlled experiments) are established methods in Software Engineering (SE) research to understand developers’ activities or the pros and cons of a technique, tool, or practice. Various guidelines and recommendations on designing and conducting different types of empirical studies in SE exist. However, the use of financial incentives (i.e., paying participants to compensate for their effort and improve the validity of a study) is rarely mentioned Objective: In this article, we analyze and discuss the use of financial incentives for human-oriented SE experimentation to derive corresponding guidelines and recommendations for researchers. Specifically, we propose how to extend the current state-of-the-art and provide a better understanding of when and how to incentivize. Method: We captured the state-of-the-art in SE by performing a Systematic Literature Review (SLR) involving 105 publications from six conferences and five journals published in 2020 and 2021. Then, we conducted an interdisciplinary analysis based on guidelines from experimental economics and behavioral psychology, two disciplines that research and use financial incentives. Results: Our results show that financial incentives are sparsely used in SE experimentation, mostly as completion fees. Especially performance-based and task-related financial incentives (i.e., payoff functions) are not used, even though we identified studies for which the validity may benefit from tailored payoff functions. To tackle this issue, we contribute an overview of how experiments in SE may benefit from financial incentivisation, a guideline for deciding on their use, and 11 recommendations on how to design them. Conclusions: We hope that our contributions get incorporated into standards (e.g., the ACM SIGSOFT Empirical Standards), helping researchers understand whether the use of financial incentives is useful for their experiments and how to define a suitable incentivisation strategy.

show abstract

Thinking Like a Developer? Comparing the Attention of Humans with Neural Models of Code

Cited by 20 publications

References 46 publications

Diet Code is Healthy: Simplifying Programs for Pre-Trained Models of Code

Diet Code is Healthy: Simplifying Programs for Pre-Trained Models of Code

Extracting Meaningful Attention on Source Code: An Empirical Study of Developer and Neural Model Code Exploration

Guidelines for using financial incentives in software-engineering experimentation

Contact Info

Product

Resources

About