Investigating probabilistic sampling approaches for large-scale surveys in software engineering

Abstract

Background: Establishing representative samples for Software Engineering surveys is still considered a challenge. Specialized literature often presents limitations on interpreting surveys' results, mainly due to the use of sampling frames established by convenience and non-probabilistic criteria for sampling from them. In this sense, we argue that a strategy to support the systematic establishment of sampling frames from an adequate source of sampling can contribute to improve this scenario. Method: A conceptual framework for supporting large scale sampling in Software Engineering surveys has been organized after performing a set of experiences on designing such strategies and gathering evidence regarding their benefits. The use of this conceptual framework based on a sampling strategy developed for supporting the replication of a survey on characteristics of agility and agile practices in software processes is depicted in this paper. Result: A professional social network (Linkedln) was established as the source of sampling and its groups of interest as the units for searching members to be recruited. It allowed to deal with a sampling frame composed by more than 110,000 members (prospective subjects) distributed over 19 groups of interest. Then, through the similarity levels observed among these groups, eight strata were organized and 7745 members were invited, from which 291 have confirmed participation and answered the questionnaire. Conclusion: The heterogeneity and number of participants in this replication contributed to improve the strength of original survey's results. Therefore, we believe the sharing of this experience, the instruments and plan can be helpful for those researchers and practitioners interested on executing large scale surveys in Software Engineering.

An Industrial Survey of Safety Evidence Change Impact Analysis Practice

“…Two reminders were posted on each group. The benefits of using LinkedIn have been discussed in the literature (e.g., [15]), and include the increase in subjects' heterogeneity, the increase in the level of confidence in the representativeness of a sample, and the possibility of reaching a population for which no centralized bodies of professionals exist.…”

Insights into the Perceived Benefits of Kanban in Software Companies: Practitioners’ Views

“…LinkedIn professional are considered groups to be a good source of data collection for researchers and practitioners from all seniority levels [32].…”

Sampling improvement in software engineering surveys

“…A recruitment plan [8] was established to answer the following question: "Which are the LinkedIn groups concerned with Software Engineering and interested in agility characteristics and practices?". To identify the available groups, it was organized a search string according the set of agile practices and characteristics presented in [6].…”

“…Section two briefly presents the study context and its hypotheses. Section three summarizes the recruitment plan, which has been fully described in a work fully devoted to it [8]. Section four presents the resulted samples, including a revision performed on the original stratification based on the obtained samples.…”

“…Continuing our line of investigation, we recently replicated a survey on characteristics of agility in software processes by performing a systematic recruitment of subjects through a relevant population retrieved from LinkedIn, presented in [8]. This survey was previously executed twice over populations established by convenience [6], with 230 invitations sent and only 25 participants in both trials.…”