<i>P</i>-Values are Random Variables

Murdoch, Duncan J.; Tsai, Yu-Ling; Adcock, James M.

doi:10.1198/000313008x332421

Cited by 112 publications

(71 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Under the null hypothesis of conditionally random assignment, we would expect the p-values of all the regressions to closely fit a U[0,1] uniform distribution with a mean of 0.5 (Murdoch et al, 2008). Figure A2 shows histograms of the p-values of all four specifications, all of which are roughly uniformly distributed.…”

Section: A3 Randomization Checkmentioning

confidence: 99%

“…Thus, for each pre-treatment characteristic, we run approximately 400 regressions. Under conditional random assignment, the p-values of the F-tests of these regressions should be uniformly distributed with a mean of 0.5 (Murdoch, Tsai, & Adcock, 2008). Furthermore, if students are randomly assigned to sections within each course, the F-test should reject the null hypothesis of no relation between section assignment and students' pretreatment characteristics at the 5 percent, 1 percent and 0.1 percent significance level in close to 5 percent, 1 percent and 0.1 percent of the cases, respectively.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Understanding Peer Effects: On the Nature, Estimation, and Channels of Peer Effects

Feld

Zölitz

2017

Journal of Labor Economics

236

145

View full text Add to dashboard Cite

This paper estimates peer effects in a university context where students are randomly assigned to sections. While students benefit from better peers on average, low-achieving students are harmed by high-achieving peers. Analyzing students' course evaluations suggests that peer effects are driven by improved group interaction rather than adjustments in teachers' behavior or students' effort. We further show, building on Angrist (2014), that classical measurement error in a setting where group assignment is systematic can lead to substantial overestimation of peer effects. With random assignment, as is the case in our setting, estimates are only attenuated.Keywords: Peer effects, higher education, measurement error, estimation bias JEL classification: I21, I24, J24 * The authors would like to thank two anonymous referees for providing valuable comments and suggestions. We would also like to thank Joshua Angrist, Sandra Black, Lex Borghans, Harold Cuffe, Gigi Foster, Andries de Grip, Monique de Haan, Thomas Dohmen, Andreas Dzemski, David Figlio, Bart Golsteyn, Jonathan Guryan, Daniel Hamermesh, Randi Hjalmarsson, Olivier Marie, Julie Moschion, Derek Stemple, Benedikt Vogt, participants at various seminars and conferences and especially Nicolás Salamanca for helpful discussions and comments. We further thank Joël Castermans, Sanne Klasen and Kim Schippers from the SBE Scheduling Department, Sylvie Kersten from the SBE Exams Office, and Jeannette Hommes and Paul Jacobs from the Educational Research and Development Department for providing data and valuable background information. We thank Sophia Wagner for providing research assistance. -1 - IntroductionThe promise of the peer effects literature is to provide policy makers with advice that can be used to increase overall performance by simply reorganizing peer groups. When looking at the by now substantial number of published articles that estimate peer effects in education, it becomes apparent that the literature has not yet delivered this promise. This can be seen, for example, in the recent review by Sacerdote (2011), who shows that size and even the sign of peer effects estimates notably differ between and even within primary, secondary and post-secondary education.One potential reason why peer effects estimates are so varied is that there are a number of social and statistical forces that lead to similar outcomes between peers, even in the absence of causal peer effects (Angrist, 2014;Manski, 1993). Two well-known challenges to the identification of peer effects are the selection and reflection problem. The selection problem states that peer groups are usually formed endogenously and that it is empirically difficult to distinguish peer effects from selection effects. The reflection problem states that it is impossible to distinguish the effect of peers on the individual from the effect of the individual on peers if both are determined simultaneously. A number of recent peer effects studies (Carrell, Fullerton, & West, 2009;Carrell, Sacerdote, & West, 201...

show abstract

Section: A3 Randomization Checkmentioning

confidence: 99%

mentioning

confidence: 99%

Understanding Peer Effects: On the Nature, Estimation, and Channels of Peer Effects

Feld

Zölitz

2017

Journal of Labor Economics

236

145

View full text Add to dashboard Cite

show abstract

“…That is, p-values are best viewed as random variables which can take on a range of values depending on the actual data (Sterling, 1959;Murdoch et al, 2008). Consequently, it is impossible to tell from the outcome of a single (published) experiment delivering a statistically significant result whether a true effect exist.…”

Section: P-values Are Random Variables and They Correspond To Standarmentioning

confidence: 99%

“…That is, researchers are interested in the post-experimental probability of H 0 and H 1 . Most probably, for the reason that researchers do not get what they really want to see (Murdoch et al, 2008 and the only parameter NHST computes is the p-value it is welldocumented (Oakes, 1986;Gliner et al, 2002;Castro Sotos et al, 2007Wilkerson and Olson, 2010;Hoekstra et al, 2014) that many, if not most researchers confuse FRP with the p-value or α and they also confuse the complement of p-value (1-p) or α (1-α) with TRP (Pollard and Richardson, 1987;Cohen, 1994). These confusions are of major portend because the difference between these completely different parameters is not minor, they can differ by orders of magnitude, the long-run FRP being much larger than the p-value under realistic conditions (Sellke et al, 2001;Ioannidis, 2005).…”

Section: Neglecting the Full Context Of Nhst Leads To Confusions Aboumentioning

confidence: 99%

When null hypothesis significance testing is unsuitable for research: a reassessment

Szűcs

Ioannidis

2016

Preprint

107

View full text Add to dashboard Cite

Null hypothesis significance testing (NHST) has several shortcomings that are likely contributing factors behind the widely debated replication crisis of (cognitive) neuroscience, psychology, and biomedical science in general. We review these shortcomings and suggest that, after sustained negative experience, NHST should no longer be the default, dominant statistical practice of all biomedical and psychological research. If theoretical predictions are weak we should not rely on all or nothing hypothesis tests. Different inferential methods may be most suitable for different types of research questions. Whenever researchers use NHST they should justify its use, and publish pre-study power calculations and effect sizes, including negative findings. Hypothesis-testing studies should be pre-registered and optimally raw data published. The current statistics lite educational approach for students that has sustained the widespread, spurious use of NHST should be phased out.

show abstract

“…. , P m is a random sample from a U (0, 1) (Sackrowitz & Samuel-Cahn 1999, Murdoch, Tsai & Adcock 2008.…”

Section: Introductionmentioning

confidence: 99%

A New Method for Detecting Significant p-values with Applications to Genetic Data

Vélez¹,

Correa²,

Arcos‐Burgos³

2014

Rev. colomb. estad.

View full text Add to dashboard Cite

A new method for detecting significant p-values is described in this paper. This method, based on the distribution of the m-th order statistic of a U (0, 1) distribution, is shown to be suitable in applications where m → ∞ independent hypothesis are tested and it is of interest for a fixed type I error probability to determine those being significant while controlling the false positives. Equivalencies and comparisons between our method and others methods based-on p-values are also established, and a graphical representation of the distribution of the test statistic is depicted for different values of m. Finally, our proposal is illustrated with two microarray data sets.Key words: Extreme values theory, p-value, Type I error probability, Multiple testing, Genetic data. ResumenSe describe una nuevo método para la detección de valores p significativos. Este método, basado en el m-ésimo estadístico de orden de la distribución U (0, 1), es adecuado en casos en los que se realizan m → ∞ pruebas de hipótesis independientes y es de interés determinar aquellas que son significativas, controlando los falsos positivos, para una probabilidad de error tipo I predeterminada. Adicionalmente, se realiza una comparación con algunas a Ph.D Scholar. E-mail: jorge.velez@anu.edu.au b Associate professor. E-mail: jccorrea@unal.edu.co c Associate professor. E-mail: mauricio.arcos-burgos@anu.edu.au 70Jorge Iván Vélez, Juan Carlos Correa & Mauricio Arcos-Burgos pruebas clásicas y se grafica la distribución del estadístico de prueba para diferentes valores de m. Finalmente se ilustra el uso de la metodología con dos conjuntos de datos provenientes de estudios con microarreglos.Palabras clave: teoría de valores extremos, valor-p, probabilidad de error tipo I, comparaciones múltiples, datos genéticos.

show abstract

P-Values are Random Variables

Cited by 112 publications

References 6 publications

Understanding Peer Effects: On the Nature, Estimation, and Channels of Peer Effects

Understanding Peer Effects: On the Nature, Estimation, and Channels of Peer Effects

When null hypothesis significance testing is unsuitable for research: a reassessment

A New Method for Detecting Significant p-values with Applications to Genetic Data

Contact Info

Product

Resources

About