The ABC of Software Engineering Research

Abstract

A variety of research methods and techniques are available to SE researchers, and while several overviews exist, there is consistency neither in the research methods covered nor in the terminology used. Furthermore, research is sometimes critically reviewed for characteristics inherent to the methods. We adopt a taxonomy from the social sciences, termed here the ABC framework for SE research, which offers a holistic view of eight archetypal research strategies. ABC refers to the research goal that strives for generalizability over Actors (A) and precise measurement of their Behavior (B), in a realistic Context (C). The ABC framework uses two dimensions widely considered to be key in research design: the level of obtrusiveness of the research and the generalizability of research findings. We discuss metaphors for each strategy and their inherent limitations and potential strengths. We illustrate these research strategies in two key SE domains, global software engineering and requirements engineering, and apply the framework on a sample of 75 articles. Finally, we discuss six ways in which the framework can advance SE research.

Soft Theory: A Pragmatic Alternative to Conduct Quantitative Empirical Studies

“…There is a rich variety in empirical research methods, each with specific strengths and weaknesses [9]. In this paper, we focus exclusively on quantitative research and data analysis.…”

“…However, this distinction leads to a reflection on conclusive claims that we are used to make related to the data we use. If we are looking at one specific organization, and are conducting solution-seeking research [9] that leads to new tools or models which are best tailored to that organization, it makes very little sense to claim for average effects. Such results may still be useful because that particular solution is likely to lead to improvements for that specific case, rather than a generic average-based one.…”

“…Studies to confirm such "hard" theories typically adopt strategies such as laboratory experiments which facilitate a high level of control and precision of measurement. Conducting field experiments in industry settings is inherently limited given that such natural settings do not facilitate the level of control that researchers would require to achieve a high level of precision of measurement, as would be possible in laboratory experiments [9]. By distinguishing hard theoretical from softtheoretical approaches, this work aims to bridge the gap between researchers' goals and practitioners' possibilities.…”

Monitoring-aware IDEs

“…To that aim, we propose three research questions: Given the complexity of simulating an environment that requires constant monitoring, such as the likes of Adyen, we opted for a ield experiment. According to Stol and Fitzgerald [48], a ield experiment refers to an experimental study conducted in a natural setting with a high degree of realism. In this strategy, the researcher manipulates some properties in the research setting with the goal of observing an efect of some kind.…”

How do Practitioners Manage Decision Knowledge during Continuous Software Engineering? (S)

“…1) Procedure: We performed a semi-structured interview study [8], which is a survey study and thus a means to perform a field study [9], [10]. We separated the study into a design and planning, data collection, and data analysis phase.…”