Operationalizing science literacy: an experimental analysis of measurement

Abstract

Inequalities in scientific knowledge are the subject of increasing attention, so how factual science knowledge is measured, and any inconsistencies in said measurement, is extremely relevant to the field of science communication. Different operationalizations of factual science knowledge are used interchangeably in research, potentially resulting in artificially comparable knowledge levels among respondents. Here, we present data from an experiment embedded in an online survey conducted in the United States (N = 1,530) that examined the distribution of factual science knowledge responses on a 3- vs. 5-point response scale. Though the scale did not impact a summative knowledge index, significant differences emerged when knowledge items were analyzed individually or grouped based on whether the correct response was “true” or “false.” Our findings emphasize the necessity for communicators to consider the goals of knowledge assessment when making operationalization decisions.

Polarized climate change beliefs: No evidence for science literacy driving motivated reasoning in a U.S. national study.

“…Lastly, the format of the response scale of science literacy may affect the relationship with motivated reasoning. This may be particularly so for recent scales that include metacognitive judgments (definitely true, likely true, etc; McKasy et al, 2020), since metacognition was shown to be comparatively inaccurate for politicized science such as climate change (Fischer et al, 2019).…”

Knowledge Acquisition in Times of the 2020 Coronavirus Pandemic: Evidence from a Four-Wave Panel Study

“…The concept of knowledge, although seemingly straightforward, has been plagued by conceptual disagreement (McKasy et al, 2020). Scholars often distinguish between denotative and connotative knowledge (Graber, 2001).…”

“…In this paper, we focus on denotative knowledge by measuring the accuracy of people's recall of number of deadly victims and recoveries from COVID-19. Taking into account that perceived and factual knowledge are oftentimes conflated (Su et al, 2014) and that people tend to overestimate their perceived knowledge (McKasy et al, 2020), factual estimates of consistently reported statistics during the pandemic may, in this case, offer the most valid operationalization of knowledge acquisition.…”

“…To be clear, people did not have to give the exact correct number to be right: we looked at the discrepancy between real numbers and people's estimates to assess levels of accuracy (i.e., lower discrepancy indicating higher accuracy). We thus deviate from approaches that let people rate statements as true or false (e.g., McKasy et al, 2020). Respondents with no numerical value (e.g., those who gave answers such as "a lot" or "not many") were treated as missing.…”