Lopsided Reasoning on Lopsided Tests and Multiple Comparisons

Abstract

For those who have not recognized the disparate natures of tests of statistical hypotheses and tests of scientific hypotheses, one-tailed statistical tests of null hypotheses such as ∂ ≤ 0 or ∂ ≥ 0 have often seemed a reasonable procedure. We earlier reviewed the many grounds for not regarding them as such. To have at least some power for detection of effects in the unpredicted direction, several authors have independently proposed the use of lopsided (also termed split-tailed, directed or one-and-a-half-tailed) tests, two-tailed tests with α partitioned unequally between the two tails of the test statistic distribution. We review the history of these proposals and conclude that lopsided tests are never justified. They are based on the same misunderstandings that have led to massive misuse of one-tailed tests as well as to much needless worry, for more than half a century, over the various so-called 'multiplicity problems'. We discuss from a neo-Fisherian point of view the undesirable properties of multiple comparison procedures based on either (i) maximum potential set-wise (or familywise) type I error rates (SWERs), or (ii) the increasingly fashionable, maximum potential false discovery rates (FDRs). Neither the classical nor the newer multiple comparison procedures based on fixed maximum potential set-wise error rates are helpful to the cogent analysis and interpretation of scientific data.

Sacrificial Pseudoreplication in LEED Cross-Certification Strategy Assessment: Sampling Structures

“…In the current study, for standard types of significance assessment, the hybrid of the Paleo-Fisherian and Neyman-Pearsonian paradigms [i.e., null hypothesis significance tests (NHST)] are replaced by a neo-Fisherian assessment, as recommended by Hurlbert and Lombardi [14,29]. The neo-Fisherian paradigm (1) does not fix α, (2) does not describe p-values as 'significant' or 'nonsignificant', (3) does not accept null hypotheses based on high p-values but only suspends judgment, (4) interprets significance tests according to "three-valued logic", and (5) presents effect size information in conjunction with significance tests.…”

Invasive plant ecosystem engineer facilitates community and trophic level alteration for brackish marsh invertebrates

“…When necessary, data were first transformed to meet the assumptions of normality. Given the strength of patterns in our data, a Bonferroni or similar correction would be unwarranted and overly conservative (Hurlbert and Lombardi 2012). Complete reporting of test statistics and p-values will enable readers to understand the tests and data as analyzed.…”

Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate

“…The first problem is that it leads to confusion over which hypotheses to include in the family of hypotheses that is used to compute an adjusted alpha level (Feise, 2002;O'Keefe, 2003O'Keefe, , 2007Trafimow & Earp, 2017). For example, a family could include all of the hypotheses in a multiway analysis of variance (ANOVA; Cramer et al, 2016), all of the hypotheses in a single study or multistudy article (sometimes called the experimentwise error rate), all of the hypotheses in a collection of articles that address the same issue, or even all of the hypotheses that have been and/or will be conducted by a specific researcher during their career (Hurlbert & Lombardi, 2012;O'Keefe, 2003O'Keefe, , 2007Trafimow & Earp, 2017). The argument that p values lose their meaning in exploratory analyses is based on the assumption that all of the hypotheses in a study or multistudy article should be included in a family.…”

“…This universal null hypothesis predicts an overall null effect for a collection of different hypotheses (e.g., all of the hypotheses that are tested in an experiment). However, many researchers have questioned the usefulness of universal null hypotheses (Armstrong, 2014;Bender & Lange, 2001;Hurlbert & Lombardi, 2012;O'Keefe, 2003;Matsunaga, 2007;Morgan, 2007;Parascandola, 2010;Perneger, 1998;Rothman, 1990;Shulz & Grimes, 2005). One key problem is that universal null hypotheses are unlikely to be associated with theoretically-meaningful alternative hypotheses.…”