An Extended Commentary on Post-publication Peer Review in Organizational Neuroscience

Abstract


 
 
 While considerable progress has been made in organizational neuroscience over the past decade, we argue that critical evaluations of published empirical works are not being conducted carefully and consistently. In this ex- tended commentary we take as an example Waldman and colleagues (2017): a major review work that evaluates the state-of-the-art of organizational neuroscience. In what should be an evaluation of the field’s empirical work, the authors uncritically summarize a series of studies that: (1) provide insufficient transparency to be clearly un- derstood, evaluated, or replicated, and/or (2) which misuse inferential tests that lead to misleading conclusions, among other concerns. These concerns have been ignored across multiple major reviews and citing articles. We therefore provide a post-publication review (in two parts) of one-third of all studies evaluated in Waldman and colleague’s major review work. In Part I, we systematically evaluate the field’s two seminal works with respect to their methods, analytic strategy, results, and interpretation of findings. And in Part II, we provide focused reviews of secondary works that each center on a specific concern we suggest should be a point of discussion as the field moves forward. In doing so, we identify a series of practices we recommend will improve the state of the literature. This includes: (1) evaluating the transparency and completeness of an empirical article before accepting its claims, (2) becoming familiar with common misuses or misconceptions of statistical testing, and (3) interpreting results with an explicit reference to effect size magnitude, precision, and accuracy, among other recommendations. We suggest that adopting these practices will motivate the development of a more replicable, reliable, and trustworthy field of organizational neuroscience moving forward.
 
 

An Extended Commentary on Post-Publication Peer Review In Organizational Neuroscience

“…However, the same level of progress has not been made with respect to careful and consistent critical evaluation of empirical works beyond the point of initial publication. The standards within psychological science (including organizational behavior research) are changing to reflect concerns over the transparency of reporting practices, appropriate use of inferential statistics, and the replicability of published findings (Cumming, 2008(Cumming, , 2014Cumming & Maillardet, 2006;Nichols et al, 2017;Nichols et al, 2016;Simmons, Nelson, & Simonsohn, 2011;Wicherts et al, 2016). In this extended commentary, we argue that scholars of organizational neuroscience are not considering these implications often enough, especially in major reviews of the literature.…”

“…However, a long-acknowledged concern in organizational behavior research is that theories based on studies with fundamental limitations can sometimes persist, propagate, and motivate organizational practice and the behavior of individuals for decades (Ghoshal, 2005;Lindebaum & Zundel, 2013). It is beyond the scope of this commentary to detail examples of studies that would exemplify best practices, but we provide some recommendations and refer multiple times to comprehensive guidelines that describe what such studies would look like (e.g., Cumming, 2008Cumming, , 2014Cumming & Maillardet, 2006;Nichols et al, 2017;Nichols et al, 2016;Simmons et al, 2011;Wicherts et al, 2016).…”

Publishing more than reviewing? Some ethical musings on the sustainability of the peer review process

“…7 serious doubts about their validity is justified. A relevant example is a recent extended postpublication commentary (Prochilo et al, 2019) on a range of so-called 'seminal' studies on organizational neuroscience (Waldman et al, 2011;Waldman et al, 2017). Prochilo and colleagues (2019) demonstrated in these studies a lack of transparency for findings to be clearly understood, evaluated, or replicated -and the misuse of inferential tests that lead to misleading conclusions.…”

“…But not only that; if we take a step back and entertain for a moment the question what happens if these deficiencies are not detected, or when opportunities for improvement are not offered in the peer-review process (due to lack of broader expertise), then we are confronted with the ethical fallout of only a limited expertise being available for the review process. A relevant example is a recent extended post-publication commentary (Prochilo et al, 2019) on a range of 'seminal' studies on organisational neuroscience (Waldman et al, 2011(Waldman et al, , 2017. Prochilo et al (2019) demonstrated that these studies lack transparency to be clearly understood, evaluated or replicated -in addition to the misuse of inferential tests that lead to misleading conclusions.…”

“…A relevant example is a recent extended post-publication commentary (Prochilo et al, 2019) on a range of 'seminal' studies on organisational neuroscience (Waldman et al, 2011(Waldman et al, , 2017. Prochilo et al (2019) demonstrated that these studies lack transparency to be clearly understood, evaluated or replicated -in addition to the misuse of inferential tests that lead to misleading conclusions. Against the backdrop of mounting concern amongst neuroscientists themselves about the validity of the ways in which neuroscientific data are applied (Button et al, 2013;Thibault and Raz, 2017;Wexler and Thibault, 2019), Button et al (2013) caution about the 'ethical dimensions' of 'unreliable research [which] is inefficient and wasteful' (p. 365).…”