A Review of Conceptual Approaches and Empirical Evidence on Probability and Nonprobability Sample Survey Research

Abstract

There is an ongoing debate in the survey research literature about whether and when probability and nonprobability sample surveys produce accurate estimates of a larger population. Statistical theory provides a justification for confidence in probability sampling as a function of the survey design, whereas inferences based on nonprobability sampling are entirely dependent on models for validity. This article reviews the current debate about probability and nonprobability sample surveys. We describe the conditions under which nonprobability sample surveys may provide accurate results in theory and discuss empirical evidence on which types of samples produce the highest accuracy in practice. From these theoretical and empirical considerations, we derive best-practice recommendations and outline paths for future research.

Does moral reasoning influence public values for health care priority setting?: A population-based randomized stated preference survey

“…Interestingly, this is the same consideration that motivated our hypothesis of increased preference for children through moral reasoning. We presumed that the ethical nuances attached to funding health interventions for children -including distinguishing features such as vulnerability, dependency, neglect, and future potential, that stood out in prior normative analyses of child health and social policy -would come through more clearly to participants when exposed to a range of principles touching on them [43][44][45][46][47][54][55][56][57][58].…”

“…In contrast to prior evidence demonstrating diminished preference for younger adults induced through moral deliberation [10], we hypothesized that a moral reasoning exercise would increase the strength of public preference for allocation to children, as compared to adults. This hypothesis was predicated on insights from foregoing normative analyses of public policies for children, which have identified distinguishing characteristics of childhood -such as vulnerability, dependency, rarity, social distinction, and future potential -as drivers for policy development [43][44][45][46][47][48]. We postulated that the moral reasoning exercise, which incorporated a number of values-based considerations specific to children, would prompt participants to consider their allocative preferences in light of these unique concerns.…”

“…Development of the survey instrument drew on prior literature on the ethics of health resource allocation and social values relevant to child health policy. The principles included in our moral reasoning exercise were derived from a systematic review of the literature on social values relevant to child health and social policy, refined through an in-depth qualitative case study of health technology assessment and policymaking for children in Canada [47,48]. Iterative refinements to the survey were informed by a pilot phase with experts (n = 3) and laypersons (n = 2) and field testing with members of the public (n = 32).…”

“…2). We fashioned principles to capture concepts identified as uniquely germane to health resource allocation involving adults and children, including 'fair innings', vulnerability, dependency, future potential, and distinction, as derived from a foregoing systematic review of social values relevant to health and social policy for children [47]. Balance was sought between principles that might inherently favour allocation to either children or adults.…”

Graphical Causal Models for Survey Inference

“…The data one actually measures in the survey are conditional on S = 1; i.e., one does not observe P (Y ), but merely P (Y |S = 1). Accordingly, sample selection is a problem because one is forced to condition on the variable S. From this perspective, unit nonresponse in a probability sample is equivalent to selection into a nonprobability sample, although the resulting biases are often larger and harder to adjust for in the latter case (Cornesse et al, 2020).…”