It depends on how you ask: measuring bias in population surveys of compliance with COVID-19 public health guidance

Abstract

ObjectiveAccurate measurement of compliance with COVID-19 guidance is important for public health policy and communications. Responses to surveys, however, are susceptible to psychological biases, including framing effects and social desirability. Our aim was to measure the effects of these biases on estimates of compliance with public health guidance (eg, hand-washing, social distancing).DesignWe conducted two online experiments (n=1800) and varied whether questions were framed positively or negatively (eg, ‘I always wash my hands…’ vs ‘I don’t always wash my hands…’). We also varied the degree to which anonymity was assured, via a ‘list’ experiment.ResultsReported compliance, despite being generally high, was reduced by negatively framing questions and increasing anonymity using a list experiment technique. Effect sizes were large: compliance estimates diminished by up to 17% points and 10% points, respectively.ConclusionEstimates of compliance with COVID-19 guidance vary substantially with how the question is asked. Standard tracking surveys tend to pose questions in ways that lead to higher estimates than alternative approaches. Experimental tests of these surveys offer public health officials greater insight into the range of likely compliance estimates to better inform policy and communications.

Mixed Emotions, but Not Positive or Negative Emotions, Facilitate Legitimate Virus-Prevention Behaviors and Eudaimonic Outcomes in the Emergence of the COVID-19 Crisis

“…Additionally, despite some inconsistencies, social desirability tended to bias self-reports towards more favorable ratings, such as higher ratings of eudaimonic functioning and legitimate behaviors. Given the strong emphasis on social responsibility in this pandemic, socially desirable tendencies may have strong biasing effects and may be important to adjust for in future pandemicrelated work (Timmons et al, 2020).…”

“…We hence examined global positive, negative, and mixed affective states to comprehensively examine emotional states and important pandemic-related outcomes, specifically legitimate, unsupported, and atypical virusprevention behaviors, and eudaimonic functioning. Furthermore, we adjusted for demographical variables that could be implicated in pandemic-related outcomes (e.g., Pasion et al, 2020), including age, gender, education level, household income, and household size, as well as social desirability, which may lead to artificial responses motivated by self-presentational concerns (Timmons et al, 2020). These controls ensure that findings for mixed emotions are not confounded by single-valenced emotions and demographical variability.…”

The role of race and scientific trust on support for COVID-19 social distancing measures in the United States

“…They find that there are no significant differences in social desirability across age, gender, or education, but do not analyze whether there may be differences across races/ethnicities. Finally, Timmons et al [ 57 ] utilizes a list experiment of online respondents that finds that, when compared to direct questions, list responses lead to a significantly lower percentage of respondents reporting social distance compliance. This would suggest that direct questions regarding individual compliance may lead to over-reporting, and there were significant differences between young/old and rural/urban respondents in levels of compliance.…”

The impact of COVID-19 lockdowns on physical domestic violence: Evidence from a list randomization experiment

“…We found one study which embedded a LE into a phone survey in a developing country, to examine the prevalence of vote buying in Tanzania ( Croke, 2017 ). Since the onset of the pandemic several studies have embedded an LE into an online survey, to test for truthful reporting of hygenic/social distancing behaviours during the pandemic ( Bowles, Larreguy, & Liu, 2020 ; Larsen, Nyrup, & Petersen, 2020 ; Timmons, McGinnity, Belton, Barjaková, & Lunn, 2020 ; Vandormael, Adam, Greuel, & Bärnighausen, 2020 ), but none thus far that have been incorporated into a phone survey.…”