Towards a framework to support large scale sampling in software engineering surveys

Mello, Rafael de; Silva, Pedro Correa da; Runeson, Per; Travassos, Guilherme Horta

doi:10.1145/2652524.2652567

Cited by 12 publications

(11 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Within this heterogeneous sample, we identified relevant evidence regarding the opinion of distinct groups of SE professionals regarding the survey context (de Mello et al 2014c). The findings and lessons acquired in the replicated studies supported the organization of a conceptual framework for supporting large scale sampling in SE surveys as initially summarized in (de Mello et al 2014d). …”

Section: Introductionmentioning

confidence: 61%

“…This work evolves the original recruitment plan (de Mello et al 2014a) and presents the first version of a Conceptual Framework proposed by de Mello et al (2014d) which is represented in Fig. 1.…”

Section: Research Design and Methodologymentioning

confidence: 99%

“…These additional criteria and examples of evaluating such sources using them can be found in (de Mello et al 2014d).…”

Section: Research Design and Methodologymentioning

confidence: 99%

“…Having the individual be approved, the individual was classified as "able" to participate in the experiment and perform the tasks. However, crowdsourcing tools typically hide relevant information regarding their populations, making unfeasible the clear characterization of the sampling frame (de Mello et al 2014d). Thus, although this kind of source of recruitment potentially contributes to increase the sample's size, little guidance is available to support the generalization of results.…”

Section: Web-based Sources For Samplingmentioning

confidence: 99%

“…As a result, the survey was answered by 291 subjects from 149 countries distributed across all continents and organized into five strata based on the reported participant's experiences and their SE main skills (de Mello et al 2014b). Within this heterogeneous sample, we identified relevant evidence regarding the opinion of distinct groups of SE professionals regarding the survey context (de Mello et al 2014c). The findings and lessons acquired in the replicated studies supported the organization of a conceptual framework for supporting large scale sampling in SE surveys as initially summarized in (de Mello et al 2014d).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Investigating probabilistic sampling approaches for large-scale surveys in software engineering

Mello

Silva

Travassos

2015

J Softw Eng Res Dev

Self Cite

View full text Add to dashboard Cite

Background: Establishing representative samples for Software Engineering surveys is still considered a challenge. Specialized literature often presents limitations on interpreting surveys' results, mainly due to the use of sampling frames established by convenience and non-probabilistic criteria for sampling from them. In this sense, we argue that a strategy to support the systematic establishment of sampling frames from an adequate source of sampling can contribute to improve this scenario. Method: A conceptual framework for supporting large scale sampling in Software Engineering surveys has been organized after performing a set of experiences on designing such strategies and gathering evidence regarding their benefits. The use of this conceptual framework based on a sampling strategy developed for supporting the replication of a survey on characteristics of agility and agile practices in software processes is depicted in this paper. Result: A professional social network (Linkedln) was established as the source of sampling and its groups of interest as the units for searching members to be recruited. It allowed to deal with a sampling frame composed by more than 110,000 members (prospective subjects) distributed over 19 groups of interest. Then, through the similarity levels observed among these groups, eight strata were organized and 7745 members were invited, from which 291 have confirmed participation and answered the questionnaire. Conclusion: The heterogeneity and number of participants in this replication contributed to improve the strength of original survey's results. Therefore, we believe the sharing of this experience, the instruments and plan can be helpful for those researchers and practitioners interested on executing large scale surveys in Software Engineering.

show abstract

Section: Introductionmentioning

confidence: 61%

Section: Research Design and Methodologymentioning

confidence: 99%

“…These additional criteria and examples of evaluating such sources using them can be found in (de Mello et al 2014d).…”

Section: Research Design and Methodologymentioning

confidence: 99%

Section: Web-based Sources For Samplingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Investigating probabilistic sampling approaches for large-scale surveys in software engineering

Mello

Silva

Travassos

2015

J Softw Eng Res Dev

Self Cite

View full text Add to dashboard Cite

show abstract

Exploring the industry's challenges in software testing: An empirical study

Garousi

Felderer

Kuhrmann

et al. 2020

J Software Evolu Process

View full text Add to dashboard Cite

ContextSoftware testing is an important and costly software engineering activity in the industry. Despite the efforts of the software testing research community in the last several decades, various studies show that still many practitioners in the industry report challenges in their software testing tasks.ObjectiveTo shed light on industry's challenges in software testing, we characterize and synthesize the challenges reported by practitioners. Such concrete challenges can then be used for a variety of purposes, eg, research collaborations between industry and academia.MethodOur empirical research method is opinion survey. By designing an online survey, we solicited practitioners' opinions about their challenges in different testing activities. Our dataset includes data from 72 practitioners from eight different countries.ResultsOur results show that test management and test automation are considered the most challenging among all testing activities by practitioners. Our results also include a set of 104 concrete challenges in software testing that may need further investigations by the research community.ConclusionWe conclude that the focal points of industrial work and academic research in software testing differ. Furthermore, the paper at hand provides valuable insights concerning practitioners' “pain” points and, thus, provides researchers with a source of important research topics of high practical relevance.

show abstract

Experimentation with dynamic simulation models in software engineering: planning and reporting guidelines

França

Travassos

2015

Empir Software Eng

Self Cite

View full text Add to dashboard Cite

Simulation-based studies (SBS) have become an interesting investigation approach for Software Engineering (SE). However, the reports on experiments with dynamic simulation models found in the technical literature lack relevant information, hampering the full understanding of the procedures and results reported, as well as their replicability. Apart from the limitations on the length in conferences and journal papers, some of the relevant information seems to be missing due to methodological issues not considered when conducting such studies. This is the case of missing research questions and goals, lack of evidence regarding the dynamic simulation model validity, poorly designed simulation experiments, amongst others. Based on findings from a previous quasi-systematic literature review, we propose a set of reporting guidelines for SBS with dynamic models in the context of SE aiming at providing guidance on which information the report should contain. Furthermore, these guidelines were evolved to support SBS planning by identifying potential threats to simulation study validity and in making recommendations to avoid them, through qualitative analysis and external evaluation. Finally, we conducted different evaluations regarding both the reporting and planning guidelines, apart from using them to support the planning of a SBS as regards software evolution. A set of 33 reporting and planning guidelines for different stages of the simulation lifecycle and focused on the experimentation with dynamic simulation models have been put together. The first assessments point to a comprehensive set of guidelines, supporting a comprehensive preparation and review of the plans and reports from the studies, apart from the planning of a SBS focused on software evolution, potentially reducing the threats to the experimentation with the validity of dynamic simulation models. The 33 guidelines cannot be understood as separate groups for reporting and planning as they overlap in many aspects. The Empir Software Eng main goal is to use the guidelines to support the planning of a simulation-based study with dynamic models so that experimenters may identify potential threats to validity and produce relevant information for a complete simulation experiment report in advance. Despite their initial contribution to increase the validity of SBS, the reporting and planning of simulationbased experiments with dynamic models still has to be discussed and improved in SE. Therefore, additional assessments of this set of guidelines are needed to strengthen the confidence in their completeness and usefulness.

show abstract

Towards a framework to support large scale sampling in software engineering surveys

Cited by 12 publications

References 13 publications

Investigating probabilistic sampling approaches for large-scale surveys in software engineering

Investigating probabilistic sampling approaches for large-scale surveys in software engineering

Exploring the industry's challenges in software testing: An empirical study

Experimentation with dynamic simulation models in software engineering: planning and reporting guidelines

Contact Info

Product

Resources

About