When Shooting for the Stars Becomes Aiming for Asterisks: P-Hacking in Family Business Research

Abstract

As a side-effect of increasing publication pressures, academics may be tempted to engage in p-hacking: a questionable research practice involving the iterative and incompletely-disclosed adjustment of data collection, analysis, and/or reporting, until nonsignificant results turn significant. Prior studies in entrepreneurship-related disciplines carry the implicit notion that p-hacking is predominantly an issue in top-tier journals, where incentives to do so may be highest. This study investigates p-hacking in the family business literature, a research field with roots in the broader entrepreneurship and small business literatures, and in which discourse increasingly takes place in both dedicated field journals and in the top-tier outlets in entrepreneurship and management. Analyses of p-values published in these field- and top-tier journals allow for a comparison of the prevalence and correlates of p-hacking at these different levels of prestige. The findings suggest that p-hacking is an issue of substantial—and statistically indistinguishable—magnitudes in both field- and top-tier journals. We further observe negative correlations of female authorship and employer prestige with p-hacking, where the latter is stronger in field versus top-tier journals. Implications of these findings, their limitations, and some suggestions going forward are discussed, with particular attention for the promise of preregistration and registered reports.

Happy Entrepreneurs? Everywhere? A Meta-Analysis of Entrepreneurship and Wellbeing

“…Any meta-analysis is aggregating findings of primary studies and, thus, might be limited by limitations of these primary studies, such as publication bias and p-hacking. While we provide tests for publication bias, one recently published paper reports p-hacking in primary family business studies (Brinkerink, 2023), whereby results are selectively published to confirm hypotheses. Notably, the interdisciplinary nature of our meta-analytic review likely helps to mitigate concerns about p-hacking.…”

Are Family Firms Doing More Innovation Output With Less Innovation Input? A Replication and Extension

“…Moreover, the range of observed effect sizes also increased indicating a substantial and increasing heterogeneity across the primary studies. 19 Finally, we also conducted a robustness check for the models in Tables 2 and 3 including only studies published in high-quality journals to check if publication practices in the family business domain (Brinkerink, 2021) have an impact on our results. 20 The results remained similar (see study Supplemental Tables C1 and C2).…”

“…9 Similar to DKEZ, we included all kinds of article types. Recent findings by Brinkerink (2021) suggest that academic journals in the family business field tend to favor statistically significant results. Relying solely on published articles could therefore cause an overall bias.…”

One finding is no finding: Toward a replication culture in family business research

“…The family business field is no exception. At the time we started developing this article, we were able to identify only three dedicated independent replications (Hisrisch & Cahill, 1995;López-Delgado & Diéguez-Soto, 2015;Weismeier-Sammer, 2011) in a literature that now easily spans thousands of publications (Rovelli, Ferasso, De Massis, & Kraus, 2021) and also is not immune to biased research practices (Brinkerink, 2021). The lack of replication culture may be particularly worrisome for family business research.…”

“…For instance, going forward one could citeDuran et al (2016) as(Duran et al, 2016; replication attempt by Block et al, 2022).16 Reviewers should be independent from the authors of the original study, obviously. An exception here may be empirical validation, where the review of validation proposals may be the optimal stage in which authors of the original study could secure conceptual validity of the proposed approach; perhaps not in a blinded way as to add transparency.17 The registered report format should improve the quality and objectivity of all forms of theory-testing in family business researchboth original and replication studiesas discussed in more detail inBrinkerink (2021). This broader discussion, although much needed, goes beyond the scope of the current article.18 Additionally, ETP offers a separate registered report format not (explicitly) targeting replication studies, such that family business scholars could presumably still consider sending in their replication plan as a registered report format.…”