Correlates of programmer efficacy and their link to experience: a combined EEG and eye-tracking study

Abstract

Background: Despite similar education and background, programmers can exhibit vast differences in efficacy. While research has identified some potential factors, such as programming experience and domain knowledge, the effect of these factors on programmers' efficacy is not well understood. Aims: We aim at unraveling the relationship between efficacy (speed and correctness) and measures of programming experience. We further investigate the correlates of programmer efficacy in terms of reading behavior and cognitive load. Method: For this purpose, we conducted a controlled experiment with 37 participants using electroencephalography (EEG) and eye tracking. We asked participants to comprehend up to 32 Java sourcecode snippets and observed their eye gaze and neural correlates of cognitive load. We analyzed the correlation of participants' efficacy with popular programming experience measures. Results: We found that programmers with high efficacy read source code more targeted and with lower cognitive load. Commonly used experience levels do not predict programmer efficacy well, but selfestimation and indicators of learning eagerness are fairly accurate. Implications: The identified correlates of programmer efficacy can be used for future research and practice (e.g., hiring). Future research should also consider efficacy as a group sampling method, rather than using simple experience measures. CCS CONCEPTS• Human-centered computing → Empirical studies in HCI ; HCI design and evaluation methods.

A Comprehensive Survey of EEG Preprocessing Methods for Cognitive Load Assessment

“…Specifically, data from one participant with fewer than 30 trials left per task after denoising with baseline correction, band-pass filtering, and ICA were excluded [138]. In a code review task, the entire data set from one participant was rejected because it contained many data points with very slow or very high response times, which was translated into the absence of code comprehension [115].…”

“…Low-pass filters [38], [104], [116], [126], [128], [133], high-pass filters [13], [38], [104], [112], [116], [123], [126]- [128], [132], [142],and band-pass filters have been constantly used [13], [14], [90], [93], [94], [96]- [101], [106], [110], [112], [115], [119]- [122], [125], [129], [130], [136]- [139], [141].High-pass cut-off frequencies were clustered in the range [0.1,1] Hz [13], [14], [38], [90], [93], [94], [96]- [100], [106], [107], [110], [116], [119]-…”

“…FIR filtering was documented in many experimental situations [14], [93], [95], [99], [100], [112], [115], [116], [125], [126], [128], [141]. IIR filtering [132], particularly a 2nd-6th order Butterworth type, was also cited [38], [97], [109], [120], [129], [130], [138], [139].…”

“…Power line interference was frequently removed from the signal [123] using Notch filter [126], [133] surrounding the frequencies of 50 Hz [38], [90], [95], [115]- [117], [135] or 60 Hz [129].The Cleanline procedure at 50 Hz and at 100 Hz was reported once [137]. ICA (see Table 6) was a popular BSS denoising approach.…”

“…In one study, artifact detection and rejection/correction were completely automated by employing the MARA approach [137]. Noisy ICs were also rejected automatically [115].ICLabel and a modified version of the BLINKER algorithm were employed [128].…”