The Long Way From α-Error Control to Validity Proper

Abstract

Several influential publications have sensitized the community of behavioral scientists to the dangers of inflated effects and false-positive errors leading to the unwarranted publication of nonreplicable findings. This issue has been related to prominent cases of data fabrication and survey results pointing to bad practices in empirical science. Although we concur with the motives behind these critical arguments, we note that an isolated debate of false positives may itself be misleading and counter-productive. Instead, we argue that, given the current state of affairs in behavioral science, false negatives often constitute a more serious problem. Referring to Wason's (1960) seminal work on inductive reasoning, we show that the failure to assertively generate and test alternative hypotheses can lead to dramatic theoretical mistakes, which cannot be corrected by any kind of rigor applied to statistical tests of the focal hypotheses. We conclude that a scientific culture rewarding strong inference (Platt, 1964) is more likely to see progress than a culture preoccupied with tightening its standards for the mere publication of original findings.

Too Good to be False: Nonsignificant Results Revisited

“…The result that 2 out of 3 papers containing nonsignificant results show evidence of at least one false negative empirically verifies previously voiced concerns about insufficient attention for false negatives (Fiedler, Kutzner, & Krueger, 2012). The Fisher test proved a powerful test to inspect for false negatives in our simulation study, where three nonsignificant results already results in high power to detect evidence of a false negative if sample size is at least 33 per result and the population effect is medium.…”

“…Cohen (1962) and Sedlmeier and Gigerenzer (1989) already voiced concern decades ago and showed that power in psychology was low. Fiedler et al (2012) contended that false negatives are harder to detect in the current scientific system and therefore warrant more concern. Despite recommendations of increasing power by increasing sample size, we found no evidence for increased sample size (see Figure 5).…”

“…Given that the complement of true positives (i.e., power) are false negatives, no evidence either exists that the problem of false negatives has been resolved in psychology. Moreover, Fiedler, Kutzner, and Krueger (2012) expressed the concern that an increased focus on false positives is too shortsighted because false negatives are more difficult to detect than false positives. They also argued that, because of the focus on statistically significant results, negative results are less likely to be the subject of replications than positive results, decreasing the probability of detecting a false negative.…”

No evidence for bilingual cognitive advantages: A test of four hypotheses.

“…Purely verbal and imprecise theories, however, ultimately become non-falsifiable as the absence of evidence can always be explained in terms of an infinite number of possible moderators suppressing or boosting the hypothesized effects. To avoid this issue, theory-driven hypotheses should predict specific situations and constellations for which bilingual advantages are expected, but also such in which no effects are expected to occur (Fiedler, Kutzner, & Krueger, 2012;see also Paap, 2014;Paap & Greenberg, 2013).…”

Barriers to Progress in Speaker Identification with Comments on the Trayvon Martin Case

“…On the other hand, Bower 6 simply identifies the problem as one resulting from "closed thinking". These two psychologists, plus others 12,15,55 , decry this situation. That is, that too many of the mainline psychological theorists dismiss or ignore the work of others as being a threat to their own 56 .…”