The nature of the relationship between diversity and stability has become the subject of intense research effort over the last few decades as the role of diversity as a major driver of ecosystem functioning and stability has come to the forefront of ecological interest. Here, we present a meta‐analysis of the impact of twelve experimental design factors on the strength and direction of relations between biotic richness and temporal variability at both the aggregate community‐ and population‐level. Based on 35 studies that report 59 community‐level and 36 population‐level relations, our results show that biotic richness has a highly general stabilizing effect on community properties that are only marginally affected by the nuances of experimental design. In contrast, experimental design factors have a highly significant effect on mean effect sizes and the resulting interpretation of relations between richness and population‐level variability. The strongest dichotomous effect was observed based on the method of calculating the response variable, such that when population variability was calculated as the mean variability of populations across all replicates, biotic richness showed a negative (stabilizing) mean effect size. In contrast, when population variability was calculated on a per replicate basis, biotic richness showed a positive (destabilizing) mean effect size. This latter result suggests that a renewed focus on the mechanisms by which populations can be stabilized (and destabilized) by diversity is needed.