The Use and Misuse of Organizational Research Methods ‘Best Practice’ Articles

Abstract

This study explores how researchers in the organizational sciences use and/or cite methodological ‘best practice’ (BP) articles. Namely, are scholars adhering fully to the prescribed practices they cite, or are they cherry picking from recommended practices without disclosing? Or worse yet, are scholars inaccurately following the methodological best practices they cite? To answer these questions, we selected three seminal and highly cited best practice articles published in Organizational Research Methods (ORM) within the past ten years. These articles offer clear and specific methodological recommendations for researchers as they make decisions regarding the design, measurement, and interpretation of empirical studies. We then gathered all articles that have cited these best practice pieces. Using comprehensive coding forms, we evaluated how authors are using and citing best practice articles (e.g., if they are appropriately following the recommended practices). Our results revealed substantial variation in how authors cited best practice articles, with 17.4% appropriately citing, 47.7% citing with minor inaccuracies, and 34.5% inappropriately citing BP articles. These findings shed light on the use (and misuse) of methodological recommendations, offering insight into how we can better improve our digestion and implementation of best practices as we design and test research and theory. Key implications and recommendations for editors, reviewers, and authors are discussed.

Literature reviews of qualitative accounting research: challenges and opportunities

“…In addition to focusing on a certain domain or method theory, Aguinis et al (2023) highlight that many impactful literature reviews of management research focus on methodological issuesjust as the present paper does [3]. Systematic "methodological literature reviews", and thus, use case (iii) for literature reviews, typically examine a specific research method applied to a certain field, systematically identify the challenges and shortcomings in its current application and often end with suggestions or best practices on how these methods should be used in the future (Aguinis et al, 2023;Kreamer et al, 2021).…”

“…In addition to focusing on a certain domain or method theory, Aguinis et al (2023) highlight that many impactful literature reviews of management research focus on methodological issues – just as the present paper does [3]. Systematic “methodological literature reviews”, and thus, use case (iii) for literature reviews, typically examine a specific research method applied to a certain field, systematically identify the challenges and shortcomings in its current application and often end with suggestions or best practices on how these methods should be used in the future (Aguinis et al , 2023; Kreamer et al , 2021). While not yet frequent, some methodological literature reviews of accounting research are available (Bedford and Speklé, 2018; Hiebl and Richter, 2018; Nitzl, 2016; van der Stede et al , 2005), and a small number of systematic reviews of qualitative methods in accounting research also exist (Dai et al , 2019; Feldermann and Hiebl, 2020; Parker and Northcott, 2016) [4].…”

Eight characteristics of rigorous multilevel implementation research: a step-by-step guide

“…Table 1 summarizes each characteristic and associated recommendations for implementation researchers; we envision it could be used as a simple planning tool or evaluative checklist. We also hope Table 1 encourages readers to use our eight characteristics as a whole, avoiding the problems associated with best practice misuse (e.g., cherry-picking specific sections to justify singular decisions while ignoring the others) [ 16 ].…”

Let’s Fix our Own Problem: Quelling the Irresponsible Research Perfect Storm

“…The concept of goal setting has long been considered important for understanding the motivational processes and behaviours that have a positive impact on performance (e.g., Colbert and Witt, 2009; Kim et al, 2018). Thus, setting goals specifically about scholars’ knowledge, skills, and abilities (KSAs) can help mitigate the growing concerns about insufficient knowledge necessary to conduct and produce responsible research (Kreamer et al, 2021; Williams et al, 2021).…”

“…Third, a salient culture and reward system exists that pressures management scholars to predominantly and almost exclusively submit their manuscripts to a small number of ‘A‐journals’ (Aguinis et al, 2020). Fourth, there are growing concerns about insufficient knowledge and training of scholars (Kreamer et al, 2021; Williams et al, 2021). Fifth, there are repeated yet unanswered calls to improve the quality and consistency of manuscript evaluations provided by journal editors and reviewers (Bedeian, 2003; Carpenter, 2009; Epstein, 1995; Lepak, 2009).…”