Sampling in Software Engineering Research: A Critical Review and Guidelines

Baltes, Sebastian; Ralph, Paul

doi:10.48550/arxiv.2002.07764

Cited by 16 publications

(29 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In a recent article, Baltes and Ralph [15] provide a primer on sampling in SE research. They define sampling as the process of selecting a smaller group of items to study, the sample, from a larger group of items of interest, the population.…”

Section: Samplingmentioning

confidence: 99%

“…Baltes and Ralph [15] identify two problems with sampling frames: first, for many software engineering phenomena there is no suitable sampling frame from which to draw a sample; second, some software engineering studies adopt poorly understood sampling strategies such as random sampling from a non-representative surrogate population. Baltes and Ralph [15] also write, "For our purposes, representativeness is the degree to which a sample's properties (of interest) resemble those of a target population." ( [15]; emphasis in original).…”

Section: Samplingmentioning

confidence: 99%

“…Baltes and Ralph [15] also write, "For our purposes, representativeness is the degree to which a sample's properties (of interest) resemble those of a target population." ( [15]; emphasis in original).…”

Section: Samplingmentioning

confidence: 99%

See 2 more Smart Citations

Recruiting credible participants for field studies in software engineering research

Rainer¹,

Wohlin²

2021

Preprint

View full text Add to dashboard Cite

Context: Software practitioners are a primary provider of information for field studies in software engineering. Research typically recruits practitioners through some kind of sampling. But sampling may not in itself recruit credible participants. Objectives: To propose and demonstrate a framework for recruiting professional practitioners as credible participants in field studies of software engineering. Method: We review existing guidelines, checklists and other advisory sources on recruiting participants for field studies. We develop a framework, partly based on our prior research and on the research of others. We search for and select three exemplar studies (a case study, an interview study and a survey study) and use those to demonstrate the application of the framework. Results: Whilst existing guidelines etc. recognise the importance of recruiting participants, there is limited guidance on how to recruit the right participants. Our demonstration of the framework with three exemplars shows that at least some members of the research community are aware of the need to carefully recruit participants. Conclusions:The framework provides a new perspective for thinking about the recruitment of credible practitioners for field studies of software engineering. In particular, the framework identifies a number of characteristics not explicitly addressed by existing guidelines.

show abstract

Section: Samplingmentioning

confidence: 99%

Section: Samplingmentioning

confidence: 99%

See 1 more Smart Citation

Recruiting credible participants for field studies in software engineering research

Rainer¹,

Wohlin²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We conduct interpretivist research as the methods of natural science are insufficient for understanding the case in its social reality context [2]. Figure 1 illustrates the context, the case under study, and the units of analysis.…”

Section: Research Design and Goalsmentioning

confidence: 99%

“…The study relies on non-probability sampling [2], i.e., there will be no element of randomness when selecting items in the sampling frame. Instead, we will use a combination of purposive and referralchain sampling to select interviewees.…”

Section: Data Collectionmentioning

confidence: 99%

Adopting Automated Bug Assignment in Practice -- A Registered Report of an Industrial Case Study

Borg,

Jonsson,

Engström

et al. 2021

Preprint

View full text Add to dashboard Cite

Background/Context] The continuous inflow of bug reports is a considerable challenge in large development projects. Inspired by contemporary work on mining software repositories, we designed a prototype bug assignment solution based on machine learning in 2011-2016. The prototype evolved into an internal Ericsson product, TRR, in 2017-2018. TRR's first bug assignment without human intervention happened in 2019. [Objective/Aim] Our exploratory study will evaluate the adoption of TRR within its industrial context at Ericsson. We seek to understand 1) how TRR performs in the field, 2) what value TRR provides to Ericsson, and 3) how TRR has influenced the ways of working. Secondly, we will provide lessons learned related to productization of a research prototype within a company. [Method] We design an industrial case study combining interviews with TRR developers and users with analysis of data extracted from the bug tracking system at Ericsson. Furthermore, we will analyze sprint planning meetings recorded during the productization. Our data analysis will include thematic analysis, descriptive statistics, and Bayesian causal analysis.

show abstract

A fine-grained data set and analysis of tangling in bug fixing commits

et al. 2022

View full text Add to dashboard Cite

Context Tangled commits are changes to software that address multiple concerns at once. For researchers interested in bugs, tangled commits mean that they actually study not only bugs, but also other concerns irrelevant for the study of bugs. Objective We want to improve our understanding of the prevalence of tangling and the types of changes that are tangled within bug fixing commits. Methods We use a crowd sourcing approach for manual labeling to validate which changes contribute to bug fixes for each line in bug fixing commits. Each line is labeled by four participants. If at least three participants agree on the same label, we have consensus. Results We estimate that between 17% and 32% of all changes in bug fixing commits modify the source code to fix the underlying problem. However, when we only consider changes to the production code files this ratio increases to 66% to 87%. We find that about 11% of lines are hard to label leading to active disagreements between participants. Due to confirmed tangling and the uncertainty in our data, we estimate that 3% to 47% of data is noisy without manual untangling, depending on the use case. Conclusion Tangled commits have a high prevalence in bug fixes and can lead to a large amount of noise in the data. Prior research indicates that this noise may alter results. As researchers, we should be skeptics and assume that unvalidated data is likely very noisy, until proven otherwise.

show abstract

Sampling in Software Engineering Research: A Critical Review and Guidelines

Cited by 16 publications

References 24 publications

Recruiting credible participants for field studies in software engineering research

Recruiting credible participants for field studies in software engineering research

Adopting Automated Bug Assignment in Practice -- A Registered Report of an Industrial Case Study

A fine-grained data set and analysis of tangling in bug fixing commits

Contact Info

Product

Resources

About