Measurement equivalence in probability and nonprobability online panels

Abstract

Nonprobability online panels are commonly used in the social sciences as a fast and inexpensive way of collecting data in contrast to more expensive probability-based panels. Given their ubiquitous use in social science research, a great deal of research is being undertaken to assess the properties of nonprobability panels relative to probability ones. Much of this research focuses on selection bias, however, there is considerably less research assessing the comparability (or equivalence) of measurements collected from respondents in nonprobability and probability panels. This article contributes to addressing this research gap by testing whether measurement equivalence holds between multiple probability and nonprobability online panels in Australia and Germany. Using equivalence testing in the Confirmatory Factor Analysis framework, we assessed measurement equivalence in six multi-item scales (three in each country). We found significant measurement differences between probability and nonprobability panels and within them, even after weighting by demographic variables. These results suggest that combining or comparing multi-item scale data from different sources should be done with caution. We conclude with a discussion of the possible causes of these findings, their implications for survey research, and some guidance for data users.

The Deficit and Contextual Models of Vaccine Hesitancy: A Test of the Mediation Paths

“…In addition, taking into consideration the context-dependence of vaccine hesitancy, validation of our findings among other populations is warranted. Another notable limitation of this study is the use of opt-in self-administered data collection, which offers limited control and may introduce selection biases and a potential lack of representativeness (Einarsson et al, 2022; C. Zhang et al, 2020).…”

Changes in Online Illegal Drug Buying during COVID-19: Assessing Effects due to a Changing Market or Changes in Strain using a Longitudinal Sample Design

“…Although online non-probability samples are generally found to be similar to random probability-based samples (MacInnis et al, 2018;Simmons & Bobo, 2015;Weinberg et al, 2014), a number of studies find that non-probability samples are less reliable than random probability-based samples (e.g., Einarsson et al, 2022;Lehdonvirta et al, 2021;Pickering & Blaszczynski, 2021). Furthermore, although the sample represented Americans with respect to age, sex, race and ethnicity, the survey's validity is vulnerable for a variety of reasons.…”

Tobacco and nicotine control policy implementation survey in the Russian regions: EPOHA-RF 2

“…В рамках разработки методологии было проведено тестирование анкеты на респондентахкурильщиках и бросивших курить, что позволило оперативно исправить ряд недочетов, внести нужные коррективы в опросник. Опросник был запро- 0,1 0,2 0,2 0,2 0,3 0,5 0,3 20-24 0,3 Преимущества поточной выборки по сравнению с панельной описаны в разделе методологии и подтверждаются исследованиями разных авторов [11,12]. Преимуществом этого метода является также тот факт, что при изучении определенного явления, частота которого не очень высока в популяции, как, например, потребления ЭСДН, нахождение лиц в готовых панелях с этими признаками, в нужных регионах, да еще с распределением в половозрастных группах как в населении, становится крайне проблематично.…”