Variable selection in Propensity Score Adjustment to mitigate selection bias in online surveys

Abstract

The development of new survey data collection methods such as online surveys has been particularly advantageous for social studies in terms of reduced costs, immediacy and enhanced questionnaire possibilities. However, many such methods are strongly affected by selection bias, leading to unreliable estimates. Calibration and Propensity Score Adjustment (PSA) have been proposed as methods to remove selection bias in online nonprobability surveys. Calibration requires population totals to be known for the auxiliary variables used in the procedure, while PSA estimates the volunteering propensity of an individual using predictive modelling. The variables included in these models must be carefully selected in order to maximise the accuracy of the final estimates. This study presents an application, using synthetic and real data, of variable selection techniques developed for knowledge discovery in data to choose the best subset of variables for propensity estimation. We also compare the performance of PSA using different classification algorithms, after which calibration is applied. We also present an application of this methodology in a real-world situation, using it to obtain estimates of population parameters. The results obtained show that variable selection using appropriate methods can provide less biased and more efficient estimates than using all available covariates.

Addressing fraudulent responses in online surveys: Insights from a web‐based participatory mapping study

“…The inclusion of these response types could potentially be achieved through various types of hierarchical modelling to account for pseudo‐replication, propensity score weighting to calibrate for nonprobability (Ferri‐García & Rueda, 2022), or multilevel regressions with poststratification adjustments (Mercer et al., 2017). However, such techniques may only be viable with larger data sets, may require extensive knowledge of model assumptions, are imperfect in reducing bias and still risk inflating covariance‐based estimates even with small proportions of invalid respondents (Copas et al., 2020; Dever et al., 2008; Guo et al., 2020; King et al., 2018).…”

“…Although the elimination of possible careless responders would result in the loss of some valuable information from our study, we deemed it appropriate to eliminate all possible fraudulent responses due to the relatively small sample size and implications that their inclusion could have on correlational statistics. Various statistical modelling techniques can assist researchers wanting to include careless responders or other fraudulent responses (Copas et al, 2020;Dever et al, 2008;Ferri-García & Rueda, 2022;Mercer et al, 2017); however, a similar level of concern should be granted to analyses considering the use of statistical adjustments. Furthermore, researchers who use these techniques should still report the presence of fraud in their surveys and the extent to which the modelling adjustments may differ with the exclusion of that potential fraud.…”

Ultrasound‐guided deployment of ProGlide™ device in transfemoral transcatheter aortic valve implantation and risk reduction of vascular complications: A propensity‐matched cohort study

“…24 Groups were compared to assess significance of differences before and after propensity score based IPTW, using measure of associations with Cramer's Phi (V) for categorical variables and Rsquared for continuous variables. 25 Mann-Whitney tests were used for continuous variables, whereas Pearson χ 2 tests were used for categorical variables, including endpoints.…”

Calibration and XGBoost reweighting to reduce coverage and non-response biases in overlapping panel surveys: application to the Healthcare and Social Survey

“…Therefore, although weighting within classes is a commonly used procedure for non-response cross-sectional and longitudinal weighting in panels, a more pragmatic alternative is to use a regression-based approach, all the more so when numerous auxiliary variables are available [18]. For this we are going to use the popular Propensity Score Adjustment (PSA) method [20,29,30] to model the probability that a unit k of the new theoretical sample s (j) responds to M j , where j = 1, ..., t , or that another unit k of the effective sample s (i) r responds to M j , where i = 1, ..., j − 1 , j = 2, ..., t , and i < j.…”

“…• Learning rate ∈ [0.001, 0.9] : the weight shrinkage applied after each boosting step. • Maximum depth ∈ [1,30] : the maximum number of splits that each tree can contain. • Minimum child weight ∈ [0, 10] : the minimum total of instance weights needed to consider a new partition.…”