Reanalyzing a Randomized Controlled Trial of Combination Antidepressant Treatment With Mirtazapine: Confidence Intervals Suggest Substantial Uncertainty

Lacasse, Jeffrey R.

doi:10.1891/1559-4343.13.2.149

Cited by 4 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors concluded that combination therapy was more effective but failed to report confidence intervals for their analyses. Lacasse (2011), reanalyzed the data and calculated confidence intervals for the reported Number Needed to Treat and argues that, in fact, there is "substantial uncertainty" (p. 152), and therefore combination therapy cannot be recommended on the basis of this study. Further, problems in statistical reporting may cause substantive claims to become unfalsifiable (see generally Popper, 1959Popper, /1968), among other issues.…”

Section: Chapter 3 Methodology Purpose and Outline Of The Studymentioning

confidence: 99%

Appraising Contemporary Social Work Research: Meta-Research on Statistical Reporting, Statistical Power, and Evidential Value

Dunleavy¹

2020

Preprint

View full text Add to dashboard Cite

Background. In recent years, the veracity of scientific findings has come under intense scrutinyin what has been called the “replication crisis” (sometimes called the “reproducibility crisis” or“crisis of confidence”). This crisis is marked by the propagation of scientific claims which weresubsequently contested, found to be exaggerated, or deemed false. The causes of this crisis aremany, but include poor research design, inappropriate statistical analysis, and the manipulationof study results. Though it is uncertain if social work is in the midst of a similar crisis, it is notunlikely, given parallels between the field and adjacent disciplines in crisis.Objective. This dissertation aims to articulate these problems, as well as foundational issues instatistical theory, in order to scrutinize statistical practice in social work research. In doing so, itparallels recent work in psychology, neuroscience, medicine, ecology, and other scientificdisciplines, while introducing a new program of meta-research to the social work profession.Method. Five leading social work journals were analyzed across a five-year period (2014-2018).In all 1,906 articles were reviewed, with 310 meeting inclusion criteria. The study was dividedinto three complementary parts. Statistical reporting practices were coded and analyzed in Part 1of the study (n = 310). Using reported sample sizes from these articles, a power survey wasperformed, in Part 2, for small, medium, and large effect sizes (n = 207). A novel statistical tool,the p-curve, was used in Part 3 to evaluate the evidential value of results from one journal(Research on Social Work Practice) and to assess for bias. Results from 39 of the 78 eligiblearticles were included in the analysis. Data and materials are available at: https://osf.io/45z3h/Results. Part 1: Notably, 86.1% of articles reviewed did not report an explicit alpha level. Apower analysis was performed in only 7.4% of articles. Use of p-values was common, beingreported in 96.8% of articles, but only 29% of articles reported them in exact form. Only 36.5%of articles reported confidence intervals; with the 95% coverage rate being the most common(reported in 31.3% of all studies). Effect sizes were explicitly reported in the results section ortables in a little more than half of articles (55.2%). Part 2: The mean statistical power for articleswas 57% for small effects, 88% for medium effects, and 95% for large effects. 61% of studiesdid not have adequate power (.80) to detect a small effect, 19% did not have adequate power todetect a medium effect, and 7% a large effect. A robustness test yielded similar but moreconservative estimates for these findings. Part 3: Both the primary p-curve and robustness testyielded right-skewed curves, indicating evidential value for the included set of results, and noevidence of bias.Conclusion. Overall, these findings provide a snapshot of the status of contemporary social workresearch. The results are preliminary but indicate areas where statistical design and reporting canbe improved in published research. The results of the power survey suggest that the field hasincreased mean statistical power compared to prior decades; though these findings are tentativeand have numerous limitations. The results of the p-curve demonstrate its potential as a tool forinvestigating bias within published research; while suggesting that the results included fromResearch on Social Work Practice have evidential value. In all this study provides a first steptowards a broader and more comprehensive assessment of the field.

show abstract

Section: Chapter 3 Methodology Purpose and Outline Of The Studymentioning

confidence: 99%

Appraising Contemporary Social Work Research: Meta-Research on Statistical Reporting, Statistical Power, and Evidential Value

Dunleavy¹

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Confidence intervals have been put forth as one solution to the problems surrounding p values (e.g., Fidler et al, 2004; Gardner & Altman, 1986; Rothman, 1978). Although potentially useful in practice (e.g., Lacasse, 2011), confidence intervals have also been difficult for researchers to interpret (Greenland et al, 2016; Hoekstra et al, 2014; Poole, 1987). A confidence interval is an interval estimate, calculated from sample data, for some population parameter (Cumming & Finch, 2005).…”

Section: Schools Of Statistical Inferencementioning

confidence: 99%

The Use and Misuse of Classical Statistics: A Primer for Social Workers

Dunleavy

Lacasse

2021

Research on Social Work Practice

Self Cite

View full text Add to dashboard Cite

In this article, we offer a primer on “classical” frequentist statistics. In doing so, we aim to (1) provide social workers with a nuanced overview of common statistical concepts and tools, (2) clarify ways in which these ideas have oft been misused or misinterpreted in research and practice, and (3) help social workers better understand what frequentist statistics can and cannot offer. We begin broadly, starting with foundational issues in the philosophy of statistics. Then, we outline the Fisherian and Neyman–Pearson approaches to statistical inference and the practice of null hypothesis significance testing. We then discuss key statistical concepts including α, power, p values, effect sizes, and confidence intervals, exploring several common misconceptions about their use and interpretation. We close by considering some limitations of frequentist statistics and by offering an opinionated discussion on how social workers may promote more fruitful, responsible, and thoughtful statistical practice.

show abstract

“…Mercer (2015) writes that editors and reviewers of academic journals play a central role in improving the rigor of academic literature and presents recommendations for the peer review process. This is an important issue that is increasingly recognized in the broader scientific community (Smith, 2006;Lacasse, 2011), and we are pleased to publish this article.…”

mentioning

confidence: 99%