Survey Data Quality in Analyzing Harmonized Indicators of Protest Behavior: A Survey Data Recycling Approach

Abstract

This article proposes a new approach to analyze protest participation measured in surveys of uneven quality. Because single international survey projects cover only a fraction of the world’s nations in specific periods, researchers increasingly turn to ex-post harmonization of different survey data sets not a priori designed as comparable. However, very few scholars systematically examine the impact of the survey data quality on substantive results. We argue that the variation in source data, especially deviations from standards of survey documentation, data processing, and computer files—proposed by methodologists of Total Survey Error, Survey Quality Monitoring, and Fitness for Intended Use—is important for analyzing protest behavior. In particular, we apply the Survey Data Recycling framework to investigate the extent to which indicators of attending demonstrations and signing petitions in 1,184 national survey projects are associated with measures of data quality, controlling for variability in the questionnaire items. We demonstrate that the null hypothesis of no impact of measures of survey quality on indicators of protest participation must be rejected. Measures of survey documentation, data processing, and computer records, taken together, explain over 5% of the intersurvey variance in the proportions of the populations attending demonstrations or signing petitions.

Measuring and Analyzing Protest Potential From a Survey Data Recycling Framework

“…Ex post survey harmonization is a useful approach to overcome limited coverage over periods and countries by creating a single data set that pools and adjusts variables from different surveys that were not designed to be compared in the first place (Granda & Blasczyk, 2016; Tomescu-Dubrow & Slomczynski, 2016). The SDR project has constructed a large data set that harmonizes the survey items about attending demonstrations and signing petitions from 19 international survey projects for 136 countries and territories from 1966 to 2013 (Slomczynski et al, 2016; Slomczynski et al, this issue). It is notable that the data set harmonizes regional surveys such as the Afrobarometer (AFB), Asian Barometer, and Latinobarometer, which cover historically underrepresented regions in protest research.…”

“…These harmonization and survey quality variables can be used as statistical controls in the multivariate analysis to account for methodological intersurvey variability (Kwak, 2020; Slomczynski & Tomescu-Dubrow, 2019). In their assessment of these measures, Slomczynski et al (2021) show that these measures account for at least 5% of the variation in past protest behavior and are not randomly distributed. It would seem to be important to control for these in analyses of these data.…”

“…The SDR version 1.1 data set provides a set of harmonization variables that capture properties of the source questions about joining demonstrations (Kołczyńska & Slomczynski, 2019; Slomczynski et al, 2016). Time span indicates whether a time frame for when a demonstration occurred falls within the respondent’s lifetime versus past 1 to 5 years (0 = past 1 to 5 years , 1 = lifetime).…”

How to Capture Global Protest Trends: Using Survey Data Recycling Data to Construct Cross-National Trends in Protest

“…We test for both the linear and quadratic forms, which have both received support in the literature. At the national survey level, we control for harmonization adjustments and survey quality, which are provided by the SDR data set and control for intersurvey variability reflecting artifacts of harmonization decisions and/or differences in survey quality (see also, Kwak, this issue; Slomczynski et al, this issue). Table 1 provides descriptive statistics for these variables.…”

“…It uses an additive scale ranging from 0 to 3 that sums three dichotomous variables that capture whether the survey has: (1) nonunique records; (2) more than 5% of missing data on either age or gender; or (3) errors in respondent IDs, which is likely linked to errors in archiving or other survey management problems. Past work has shown that these errors can possibly lead to distorted empirical results (Slomczynski et al, this issue; Slomczynski, Powałko, et al, 2017; Zieliński et al, 2018). Seventeen percent of our source files had one or more of these problems.…”