No abstract

Skilled workers are crucial to the success of software development. The current practice in research and industry for assessing programming skills is mostly to use proxy variables of skill, such as education, experience, and multiple-choice knowledge tests. There is as yet no valid and efficient way to measure programming skill. The aim of this research is to develop a valid instrument that measures programming skill by inferring skill directly from the performance on programming tasks. Over two days, 65 professional developers from eight countries solved 19 Java programming tasks. Based on the developers' performance, the Rasch measurement model was used to construct the instrument. The instrument was found to have satisfactory (internal) psychometric properties and correlated with external variables in compliance with theoretical expectations. Such an instrument has many implications for practice, for example, in job recruitment and project allocation.

This study investigates the role of working memory and experience in the development of programming knowledge and programming skill. An instrument for assessing programming skill – where skill is inferred from programming performance – was administered along with tests of working memory and programming knowledge. We recruited 65 professional software developers from nine companies in eight European countries to participate in a 2-day study. Results indicate that the effect of working memory and experience on programming skill is mediated through programming knowledge. Programming knowledge was further found to explain individual differences in programming skill to a large extent. The overall findings support Cattell’s investment theory. Further, we discuss how this study, which currently serves a pilot function, can be extended in future studies. Although low statistical power is a concern for some of the results reported, this work contributes to research on individual differences in high-realism work settings with professionals as subjects.

Skilled developers are important to the software industry. In empirical studies in software engineering, knowing the skill level of the participants is also important for correct interpretation of results. The current practice in industry and research for assessing programming skills is mostly to use proxy variables of skill, such as education, experience, and multiple-choice knowledge tests. There is as yet no valid and e cient way to measure programming skill. Consequently, this thesis aimed to construct a valid instrument for measuring programming skill, where skill is inferred from performance on programming tasks. The Rasch measurement model was used to construct the instrument. Sixty-five professional developers from eight countries participated in validating the instrument, solving 19 Java programming tasks over two days. The validity of the instrument was theoretically investigated through commercial and research-based tests. Programming skill, as measured by the instrument, was also investigated in terms of experience and other background variables. The instrument was found to have desirable psychometric properties, and the overall results appear well aligned with theoretical expectations. This work has shown that acceptable measures of programming skill may be obtained with less than one day of testing. Further work should be directed at reducing the time needed to measure programming skill without a↵ecting the validity of the instrument. The results of the research have already been transferred to the industry through a commercial prototype. iii First and foremost, I am profoundly thankful to my principal supervisor Dag Sjøberg. He has provided me with unwavering support during my research for this thesis, often at odd hours and in strange places. I feel privileged to have had the opportunity to occupy so much of Dag's time over the years. The making of this thesis has been a truly rewarding experience, both professionally and personally, because of him. I am also grateful to my second supervisor, Tore Dybå, for interesting discussions and for inspiring me to look to other research fields for solutions to my research problem. Other researchers have also been important at various stages. I thank Erik Arisholm for his guidance and flexibility during the early years of my work. A special thanks goes to Jo Hannay and Magne Jørgensen, who are highly inspiring researchers. I am also grateful to Jan-Eric Gustafsson for his interest in my work. This thesis would have been impossible without financial support. Simula Research Laboratory gave me the opportunity to carry out the research and Simula Innovation and Simula School of Research and Education provided support as well. The FORNY program at the Norwegian Research Council funded parts of my work, while the University of Oslo allowed me to finish the thesis at a pace that was rewarding for me. This thesis would similarly have been impossible without technical support. Early on, Steinar Haugen shared his passion for programming with me and has since become a...