Statistical judgments are influenced by the implied likelihood that samples represent the same population

Abstract

When sample information is combined, it is generally considered normative to weight information based on larger samples more heavily than information based on smaller samples. However, if samples appear likely to have been drawn from different subpopulations, it is reasonable to combine estimates of these subpopulation means (typically, the sample means) without weighting these estimates by sample size. This study investigated whether laypeople are influenced by the likelihood of samples coming from the same population when determining how to combine information. In two experiments we show that (1) implied binomial variability affected participants' judgments of the likelihood that a sample was drawn from a given population, (2) participants' judgments were more affected by sample size when samples were implied to be drawn randomly from a general population, compared to when they were implied to be drawn from different subpopulations, and (3) people higher in numeracy gave more normative responses. We conclude that when determining how to weight and combine samples, laypeople use not only the provided data, but also information about likelihood and sampling processes that these data imply.

Sample representativeness affects whether judgments are influenced by base rate or sample size

“…However, when base rates are given and relevant they should be incorporated into judgments. Chesney and Obrecht's (2012) findings do not distinguish whether their participants were attempting to use base rates (with 50% being the "best guess") in their judgments, or merely disregarding sample size information when sampling was questionable. With the current study, we further investigate this topic, examining two main questions.…”

“…After completing the 64 scenarios, participants' numerical abilities were assessed via a multiple choice version of the Lipkus, Samsa, and Rimer (2001) 10 question numerical literacy measure that has been used in previous studies (Chesney & Obrecht, 2012). Information regarding the participants' demographics, as well as their educational background was also collected at this time.…”

“…Following this line of investigation, Chesney and Obrecht (2012) found that people are less likely to weight data by sample size when samples are implied to have come from different subgroups within a population. The authors manipulated the likelihood of samples representing the same population both verbally and numerically.…”

“…Second, if base rate data are used, are they integrated into judgments only when they are relevant to the inference? Further, participants higher in numerical ability (Chapman & Liu, 2009) and with higher levels of education (Bramwell, West, & Salmon, 2006) have previously demonstrated superior Bayesian reasoning performance, and more numerate individuals show greater use of numeric information when integrating sample data (Chesney & Obrecht, 2012). Thus, we also investigated whether individual differences in numeracy correlate with use of base rates and sample size when integrating information.…”

“…Accordingly, participants' percent estimates were scaled to indicate the extent to which they were consistent with the sample size weighted mean versus the unweighted mean, via a method previously used by Chesney and Obrecht (2012). Because the predictions of the sample size weighted and base rate weighted averaging models were systematically related, as we describe below, this scaling allowed us to evaluate the predictions of all three models at once.…”