How Do Older Adults Recruited Using MTurk Differ From Those in a National Probability Sample?

Abstract

A growing number of studies within the field of gerontology have included samples recruited from Amazon’s Mechanical Turk (MTurk), an online crowdsourcing portal. While some research has examined how younger adult participants recruited through other means may differ from those recruited using MTurk, little work has addressed this question with older adults specifically. In the present study, we examined how older adults recruited via MTurk might differ from those recruited via a national probability sample, the Health and Retirement Study (HRS), on a battery of outcomes related to health and cognition. Using a Latin-square design, we examined the relationship between recruitment time, remuneration amount, and measures of cognitive functioning. We found substantial differences between our MTurk sample and the participants within the HRS, most notably within measures of verbal fluency and analogical reasoning. Additionally, remuneration amount was related to differences in time to complete recruitment, particularly at the lowest remuneration level, where recruitment completion required between 138 and 485 additional hours. While the general consensus has been that MTurk samples are a reasonable proxy for the larger population, this work suggests that researchers should be wary of overgeneralizing research conducted with older adults recruited through this portal.

Measurement Invariance of Social Media Use in Younger and Older Adults and Links to Socioemotional Health

“…Older adults who use MTurk may be more tech-savvy and may be fundamentally different from the typical older adult population due to the gray digital divide, which could limit the generalizability of the current results. Indeed, some evidence comparing older adults on MTurk and older adults from the Health and Retirement Study (HRS) found that older adults recruited from MTurk were younger, more educated, wealthier, and more likely to be female ( Ogletree & Katz, 2020 ). Additionally, MTurk older adults were more likely to have higher performance on verbal analogies and verbal fluency than HRS older adults which was partly attributable to these sociodemographic differences ( Ogletree & Katz, 2020 ).…”

“…Indeed, some evidence comparing older adults on MTurk and older adults from the Health and Retirement Study (HRS) found that older adults recruited from MTurk were younger, more educated, wealthier, and more likely to be female ( Ogletree & Katz, 2020 ). Additionally, MTurk older adults were more likely to have higher performance on verbal analogies and verbal fluency than HRS older adults which was partly attributable to these sociodemographic differences ( Ogletree & Katz, 2020 ). Therefore, the current results may not be generalizable to the larger older adult population.…”

Older Adults’ Emotion Recognition Ability Is Unaffected by Stereotype Threat

“…Generally, the limited existing research suggests that data obtained using MTurk samples are comparable to data obtained from the general population, including for older adults (e.g., Lemaster et al, 2015;Mortensen and Hughes, 2018). However, a very recent study found that, compared to a sample of the general older adult population, older adults recruited via MTurk have higher cognitive functioning, self-rated memory, and self-rated health (Ogletree and Katz, 2020). Thus, it is possible that the current studies' use of participants from MTurk may limit the generalisability of the current findings to the general population.…”

Conducting Quantitative Research with Hard-To-Reach-Online Populations: Using Prime Panels to Rapidly Survey Older Adults During a Pandemic

“…Although MTurk and similar platforms help overcome the homogeneity of university participant pools, they tend to yield samples significantly younger and less racially diverse than the American population [ 2 ]. This limitation is exacerbated by the fact that Amazon does not publish an age-based breakdown of MTurk workers [ 18 ], and that the number of MTurk workers available for a study at any given time is well below the number registered on the platform [ 19 – 21 ]. The relatively small number of active MTurk workers increases the chance that they are familiar with the methods that researchers typically use to ensure data quality [ 1 ].…”

“…An overarching concern with online research is that recruitment on crowdsourcing platforms may be undermined by bots or malicious actors misrepresenting age, location, or other characteristics, in an effort to obtain tasks and payments [ 18 , 22 ]. In addition to these concerns, survey studies on online crowdsourcing platforms are vulnerable to response satisficing, with participants paying inadequate attention while they take surveys, thus threatening data validity [ 23 ].…”