Misconceptions About Sample Size, Statistical Significance, and Treatment Effect

"But you don't look Puerto Rican": The moderating effect of ethnic identity on the relation between skin color and self-esteem among Puerto Rican women.

“…For example, this sample was substantially more educated than most mainland Puerto Ricans. Yet, despite its limited power and the restriction of range in many of the study variables (i.e., self-esteem, ethnic identity, and linguistic fluency), a large percentage of the variance in self-esteem was still able to be accounted for (Wilkerson & Olson, 1997). In addition, it is notable that a moderator effect was even found given the general difficulty of detecting these effects (Jaccard & Wan, 1995).…”

Public Administration Research and Practice: A Methodological Manifesto

“…In a more recent study, Wilkerson and Olson (1997) asked 52 psychology graduate students to evaluate two tests which report p-values of 0.05 and are identical in every way except that one has a sample size of 25 and the other has a sample size of 250. The graduate students were asked which test had the greatest probability of making a Type I error.…”

When null hypothesis significance testing is unsuitable for research: a reassessment

“…Better Training and Better Use of More Statistical Methods: from Believers to Thinkers A core problem seems to be that the statistical subject knowledge of many researchers in biomedical and social science has been shown to be poor (Oakes, 1986;Gliner et al, 2002;Castro Sotos et al, 2007Wilkerson and Olson, 2010;Hoekstra et al, 2014). NHST perfectly fits with poor understanding because of the perceived simplicity of interpreting its outcome: is p ≤ 0.05 (Cohen, 1994)?…”

“…That is, researchers are interested in the post-experimental probability of H 0 and H 1 . Most probably, for the reason that researchers do not get what they really want to see (Murdoch et al, 2008 and the only parameter NHST computes is the p-value it is welldocumented (Oakes, 1986;Gliner et al, 2002;Castro Sotos et al, 2007Wilkerson and Olson, 2010;Hoekstra et al, 2014) that many, if not most researchers confuse FRP with the p-value or α and they also confuse the complement of p-value (1-p) or α (1-α) with TRP (Pollard and Richardson, 1987;Cohen, 1994). These confusions are of major portend because the difference between these completely different parameters is not minor, they can differ by orders of magnitude, the long-run FRP being much larger than the p-value under realistic conditions (Sellke et al, 2001;Ioannidis, 2005).…”