Prior research experience is widely considered by graduate school admissions committees in the United States of America. Here, we use meta‐analytic methods and data from 18 unique samples and a total sample size of 3,525 students to shed light on the validity of prior research experience as a predictor of graduate school performance. Prior research experience was largely unrelated to academic performance (ρ = .01, k = 8, N = 1,419), degree attainment (ρ = .05, k = 3, N = 140), professional/practice performance (ρ = .06, k = 4, N = 1,120), and publication performance (ρ = .11, k = 7, N = 1,094). We also discuss whether consideration of prior research experience may unfairly disadvantage the students with lower levels of SES, students with childcare or eldercare responsibilities, and students from institutions at which research opportunities are limited.

We present the results of a meta-analytic synthesis of the literature on the association between the use of frequent low-stakes quizzes in real classes and students' academic performance in those classes. Data from 52 independent samples from real classes (N=7,864) suggests a moderate association of d=.42 between the use of quizzes and academic performance. Effects are even stronger in psychology classes (d=.47) and when quiz performance contributed to class grades (d=.51). We also find that performance on quizzes is strongly correlated with academic performance (k=19, N=3,814, r=.57) such that quiz performance is relatively strongly predictive of later exam performance. We also found that the use of quizzes is associated with a large increase in the odds of passing a class (k=5, N=1,004, odds ratio=2.566).

We examine the predicted replicability of experimental research on system justification theory (SJT) by conducting a z-curve analysis. Z-curve is a meta-analytic technique similar to p-curve, but which performs better under conditions of heterogeneity. It estimates the predicted replication rate, average power, false discovery risk, and file drawer ratio (FDR) of a sample of studies. The z-curve based on 116 papers and 232 unique samples suggests that the experimental SJT literature is likely to show low rates of replicability, as indicated by an overall average statistical power of 16%. Moderator analyses suggest that this may be driven in part by publication pressures, that the replicability of research in this area has improved since 2015, and that studies using system threat manipulations show particularly low estimated replication rates (ERR). Implications for the replicability and validity of the experimental SJT literature are discussed, and recommendations to increase the rigor of research are put forth.

The purpose of the current work was to examine the evidentiary value of the studies that have been included in published meta-analyses as a way of investigating the evidentiary value of the meta-analyses themselves. The studies included in 25 meta-analyses published in the last 10 years in Psychological Bulletin that investigated experimental mean differences were z-curved. Z-curve is a meta-analytic technique that allows one to estimate the predicted replicability, average power, publication bias, and false discovery rate of a population of studies. The results of the z-curves estimated a substantial file drawer in three-quarters of the meta-analyses; and in one-third of the meta-analyses, up to half of the studies are not expected to replicate and up to one-fifth of the studies included could be false positives. Possible reasons for these findings are discussed, and caution in interpreting published meta-analyses is recommended.

No abstract