Do you Really Code? Designing and Evaluating Screening Questions for Online Surveys with Programmers

Danilova, Anastasia; Naiakshina, Alena; Horstmann, Stefan; Smith, Matthew

doi:10.1109/icse43902.2021.00057

Cited by 40 publications

(19 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To do that, we used a one-minute timeboxed three-question multiple-choice survey. Danilova et al developed a list of the most compelling question to ask in such pilot studies; we strongly recommend using those [3].…”

Section: Selection Processmentioning

confidence: 99%

See 1 more Smart Citation

Recruiting Software Engineers on Prolific

Russo¹

2022

Preprint

View full text Add to dashboard Cite

Recruiting participants for software engineering research has been a primary concern of the human factors community. This is particularly true for quantitative investigations that require a minimum sample size not to be statistically underpowered. Traditional data collection techniques, such as mailing lists, are highly doubtful due to self-selection biases. The introduction of crowdsourcing platforms allows researchers to select informants with the exact requirements foreseen by the study design, gather data in a concise time frame, compensate their work with fair hourly pay, and most importantly, have a high degree of control over the entire data collection process. This experience report discusses our experience conducting sample studies using Prolific, an academic crowdsourcing platform. Topics discussed are the type of studies, selection processes, and power computation.

show abstract

Section: Selection Processmentioning

confidence: 99%

“…In this exposition, we will not consider construct validity issues 2. Prolific is an academic data collection platform: www.prolific.co 3. www.qualtrics.com.…”

mentioning

confidence: 99%

Recruiting Software Engineers on Prolific

Russo¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Especially the number of professional reversers is rather small and becomes even smaller when only considering the ones willing to spend time on a user survey. Additionally, recent research indicates that even popular paid services claiming to provide qualified survey participants can and should not be used to overcome this limitation [9]. Consequently, like previous studies with a similar topic [31], [47], we have to work with a relatively limited number of participants compared to less-technical user surveys.…”

Section: Survey Limitations and Threats To Validitymentioning

confidence: 99%

dewolf: Improving Decompilation by leveraging User Surveys

Enders¹,

C.²,

Bergmann³

et al. 2022

Preprint

View full text Add to dashboard Cite

Analyzing third-party software such as malware or firmware is a crucial task for security analysts. Although various approaches for automatic analysis exist and are the subject of ongoing research, analysts often have to resort to manual static analysis to get a deep understanding of a given binary sample. Since the source code of encountered samples is rarely available, analysts regularly employ decompilers for easier and faster comprehension than analyzing a binary's disassembly.In this paper, we introduce our decompilation approach dewolf. We developed a variety of improvements over the previous academic state-of-the-art decompiler and some novel algorithms to enhance readability and comprehension, focusing on manual analysis. To evaluate our approach and to obtain a better insight into the analysts' needs, we conducted three user surveys. The results indicate that dewolf is suitable for malware comprehension and that its output quality noticeably exceeds Ghidra and Hex-Rays in certain aspects. Furthermore, our results imply that decompilers aiming at manual analysis should be highly configurable to respect individual user preferences. Additionally, future decompilers should not necessarily follow the unwritten rule to stick to the code-structure dictated by the assembly in order to produce readable output. In fact, the few cases where dewolf already cracks this rule lead to its results considerably exceeding other decompilers. We publish a prototype implementation of dewolf and all survey results on GitHub [15], [40].

show abstract

“…To ensure the quality of the data, we added competence screening questions and random attention checks. The competence screening part included questions from Danilova et al [78] and aimed to filter out people who do not meet the software engineering knowledge requirement. Attention checks were randomly inserted between questions to ensure the reliability of the answers.…”

Section: Survey Data Collectionmentioning

confidence: 99%

The Impact of Working From Home on the Success of Scrum Projects: A Multi-Method Study

Cucolas¹,

Russo²

2021

Preprint

View full text Add to dashboard Cite

The number of companies opting for remote working has been increasing over the years, and Agile methodologies, such as Scrum, were adapted to mitigate the challenges caused by the distributed teams. However, the COVID-19 pandemic imposed a fully working from home context, which has never existed before. This paper investigation a two-phased Multi-Method study. In the first phase, we uncover how working from home impacted Scrum practitioners through a qualitative survey. Then, in the second phase, we propose a theoretical model that we test and generalize using Partial Least Squares -Structural Equation Modeling (PLS-SEM) through a sample study of 200 software engineers who worked from home within Scrum projects. From assessing our model, we can conclude that all the latent variables are reliable and all the hypotheses are significant. Moreover, we performed an Importance-Performance Map Analysis (IPMA), highlighting the benefits of the home working environment and the use of Scrum for project success. We emphasize the importance of supporting the three innate psychological needs of autonomy, competence, and relatedness in the home working environment. We conclude that the home working environment and the use of Scrum both contribute to project success, with Scrum acting as a mediator.

show abstract

Do you Really Code? Designing and Evaluating Screening Questions for Online Surveys with Programmers

Cited by 40 publications

References 38 publications

Recruiting Software Engineers on Prolific

Recruiting Software Engineers on Prolific

dewolf: Improving Decompilation by leveraging User Surveys

The Impact of Working From Home on the Success of Scrum Projects: A Multi-Method Study

Contact Info

Product

Resources

About