Reconsidering the Role of Research Method Guidelines for Qualitative, Mixed-methods, and Design Science Research

Abstract

Guidelines for different qualitative research genres have been proposed in information systems (IS). As these guidelines are outlined for conducting and evaluating good research, studies may be denied publication simply because they do not follow a prescribed methodology. This can result in "checkbox" compliance, where the guidelines become more important than the study. We argue that guidelines can only be used to evaluate what good research is if there is evidence that they lead to certain good research outcomes. Currently, the guidelines do not present such evidence. Instead, when it is presented, the evidence is often an authority argument or evidence of popularity with usability examples. We further postulate that such evidence linking guidelines and outcomes cannot be presented. Therefore, it may be time for the IS research community to acknowledge that many research method principles we regard as authoritative may ultimately be based on speculation and opinion, and thus, they should be taken less seriously as absolute guidelines in the review process.

Toward More Rigorous Blockchain Research: Recommendations for Writing Blockchain Case Studies

“…However, the structure I provide in this paper, along with suggestions on how to incorporate theory and ensure validity, is not meant as an exhaustive checklist to be used by reviewers in order to assess the overall quality of a publication. In other words, it should not be the case that the guidelines "become more important than the study" (Holtkamp et al, 2019) or that the paper is "written according to a 'formula"' (Avison et al, 2017, p. 271). Instead, I concur with Klein and Myers (1999), p. 78) who write "while we believe that none of our principles should be left out arbitrarily, researchers need to work out themselves how (and which of) the principles apply in any particular situation."…”

Workshops as a Research Method: Guidelines for Designing and Evaluating Artifacts Through Workshops

“…Recently, some criticism against research methods guidelines or principles has been raised, because such guidelines could be misused as some sort of checklist to reject paper submissions, in cases where the respective guidelines were not sufficiently met [29]. We agree, that the relevance of a paper's content should be more important than the strict following of research method guidelines, but still we argue that such guidelines can help to maintain a rigorous research process and to enhance comparability.…”

“…Finally, we reviewed various sources about research methods in general and mixed methods and triangulation in particular [5,12,29,31,56], in order to identify relevant methods for workshop evaluations. The results of this step are described in Section 4.1.…”

“…This section summarizes the developed workshop evaluation guidelines. Following Holtkamp et al [29], we use the term "guidelines" for the overall set of different principles and tools for evaluating workshops. More specifically, we present several workshop evaluation "principles" (Section 4.1), workshop evaluation "goals" (Section 4.2), workshop evaluation "methods" (Section 4.3) and a workshop evaluation "framework" that juxtaposes appropriate methods for specific goals.…”

“…We agree, that the relevance of a paper's content should be more important than the strict following of research method guidelines, but still we argue that such guidelines can help to maintain a rigorous research process and to enhance comparability. Consequently, we follow Holtkamp et al [29] by suggesting that our developed guidelines are supposed to help researchers with their workshop evaluations but should not be used as a mere checklist without reflecting upon their appropriateness. Although we argue that our guidelines will ensure a higher standard of workshop evaluations, we acknowledge that there might be reasons for not following them strictly, such as limitations of resources or the fact that only the researchers would be able to also act as facilitators of the workshop for reasons of unique expertise.…”