Appraising Contemporary Social Work Research: Meta-Research on Statistical Reporting, Statistical Power, and Evidential Value

Abstract

Background. In recent years, the veracity of scientific findings has come under intense scrutinyin what has been called the “replication crisis” (sometimes called the “reproducibility crisis” or“crisis of confidence”). This crisis is marked by the propagation of scientific claims which weresubsequently contested, found to be exaggerated, or deemed false. The causes of this crisis aremany, but include poor research design, inappropriate statistical analysis, and the manipulationof study results. Though it is uncertain if social work is in the midst of a similar crisis, it is notunlikely, given parallels between the field and adjacent disciplines in crisis.Objective. This dissertation aims to articulate these problems, as well as foundational issues instatistical theory, in order to scrutinize statistical practice in social work research. In doing so, itparallels recent work in psychology, neuroscience, medicine, ecology, and other scientificdisciplines, while introducing a new program of meta-research to the social work profession.Method. Five leading social work journals were analyzed across a five-year period (2014-2018).In all 1,906 articles were reviewed, with 310 meeting inclusion criteria. The study was dividedinto three complementary parts. Statistical reporting practices were coded and analyzed in Part 1of the study (n = 310). Using reported sample sizes from these articles, a power survey wasperformed, in Part 2, for small, medium, and large effect sizes (n = 207). A novel statistical tool,the p-curve, was used in Part 3 to evaluate the evidential value of results from one journal(Research on Social Work Practice) and to assess for bias. Results from 39 of the 78 eligiblearticles were included in the analysis. Data and materials are available at: https://osf.io/45z3h/Results. Part 1: Notably, 86.1% of articles reviewed did not report an explicit alpha level. Apower analysis was performed in only 7.4% of articles. Use of p-values was common, beingreported in 96.8% of articles, but only 29% of articles reported them in exact form. Only 36.5%of articles reported confidence intervals; with the 95% coverage rate being the most common(reported in 31.3% of all studies). Effect sizes were explicitly reported in the results section ortables in a little more than half of articles (55.2%). Part 2: The mean statistical power for articleswas 57% for small effects, 88% for medium effects, and 95% for large effects. 61% of studiesdid not have adequate power (.80) to detect a small effect, 19% did not have adequate power todetect a medium effect, and 7% a large effect. A robustness test yielded similar but moreconservative estimates for these findings. Part 3: Both the primary p-curve and robustness testyielded right-skewed curves, indicating evidential value for the included set of results, and noevidence of bias.Conclusion. Overall, these findings provide a snapshot of the status of contemporary social workresearch. The results are preliminary but indicate areas where statistical design and reporting canbe improved in published research. The results of the power survey suggest that the field hasincreased mean statistical power compared to prior decades; though these findings are tentativeand have numerous limitations. The results of the p-curve demonstrate its potential as a tool forinvestigating bias within published research; while suggesting that the results included fromResearch on Social Work Practice have evidential value. In all this study provides a first steptowards a broader and more comprehensive assessment of the field.

The Use and Misuse of Classical Statistics: A Primer for Social Workers

“…If we are to use frequentist statistics at all, we may as well do so as rigorously as possible. This would entail improving how such concepts are taught to students (e.g., Cassidy et al, 2019; Dunleavy & Lacasse, 2020) and assessing how they are being interpreted by practicing researchers (Gigerenzer, 2018; Hoekstra et al, 2014; Oakes, 1986). If nothing else, improving our understanding of the meaning of frequentist concepts, acknowledging the numerous analytic decisions made along the way (Berk, 1977; Gelman, 2016; Simmons et al, 2011), and moving away from the reflexive dichotomization of results (McShane & Gal, 2017) should increase our understanding of these statistical concepts.…”

“…Journal editors, funders, and institutional leadership can encourage practical changes to the way we conduct, evaluate, report, and disseminate our research (Dunleavy, 2021, 2020; Rothman, 1998; Shlonsky et al, 2002; Smaldino & McElreath, 2016). Researchers could be encouraged to routinely (1) engage in sample size planning; (2) perform a priori power analyses; (3) preregister statistical analyses; (4) submit registered reports; (5) make code, analyses, and collect data available in a public, online repository (e.g., discussed by Chambers, 2017); and (6) explicitly consider the potential role of theory for the phenomenon under study (e.g., Borsboom, 2013; Gambrill, 1997, 1999; Gomory, 2001a, 2001b; Muthukrishan & Henrich, 2019; Nugent, 1987; see Dunleavy, 2020a, especially chap. 1, for a more robust discussion of such reform efforts).…”

“…For instance, when used judiciously, statistical analyses can provide a measure of fit for competing models (e.g., Glover & Dixon, 2004) or a measure of evidence for a given hypothesis (e.g., Dienes, 2016; Morey et al, 2016). Misunderstood or used inappropriately (Greenland et al, 2016), researchers can arrive at erroneous conclusions, waste finite resources, and pursue fruitless lines of research (see generally Colling & Szűcs, 2018; Dunleavy, 2020a; Ioannidis, 2005, 2012, 2016; Wagenmakers et al, 2011). Still further, incomplete or inaccurate reporting and interpretation of statistical concepts may mislead readers (Altman, 1994), resulting in inappropriate conclusions about the rigor of study design and/or the validity of study results (Glasziou et al, 2014).…”

“…This primer begins broadly, starting with foundational issues in the philosophy of statistics and ending more narrowly, emphasizing applied issues in statistical theory and practice. This is done deliberately as foundational issues in statistical inference have profound consequences for their practical application, despite their seeming absence from articles and texts on social work research (Lewis, 2019, being a rare but underappreciated exception; see also Dunleavy, 2020a). First, we briefly outline the major schools of statistical inference, giving the majority of our attention to classical statistics, more accurately described as frequentist statistics.…”

“…Along the way, we will clarify several common misconceptions about their use and interpretation. This section will be of particular interest to social work researchers and students, given the field’s historic (Cowger, 1984; LeCroy, 1985; Orme & Combs-Orme, 1986; Rosen et al, 1999; Tripodi, 1974) and continued (e.g., Dunleavy, 2020a) misuse and misreporting of statistical concepts and results. We close by considering some limitations of frequentist statistics, as used in NHST, and by considering alternative approaches to statistical practice for social work research and education.…”