Beyond Likert ratings: Improving the robustness of developmental research measurement using best–worst scaling

Abstract

Some of the ‘best practice’ approaches to ensuring reproducibility of research can be difficult to implement in the developmental and clinical domains, where sample sizes and session lengths are constrained by the practicalities of recruitment and testing. For this reason, an important area of improvement to target is the reliability of measurement. Here we demonstrate that best–worst scaling (BWS) provides a superior alternative to Likert ratings for measuring children’s subjective impressions. Seventy-three children aged 5–6 years rated the trustworthiness of faces using either Likert ratings or BWS over two sessions. Individual children’s ratings in the BWS condition were significantly more consistent from session 1 to session 2 than those in the Likert condition, a finding we also replicate with a large adult sample (N = 72). BWS also produced more reliable ratings at the group level than Likert ratings in the child sample. These findings indicate that BWS is a developmentally appropriate response format that can deliver substantial improvements in reliability of measurement, which can increase our confidence in the robustness of findings with children.

Social valuation of biodiversity relative to other types of assets at risk in wildfire

“…We opted for the BWS method because it has many advantages over conventional ranking and rating measurements. For example, BWS provides more reliable and reproducible results than Likert ratings (Burton et al., 2021). Results are also of better quality because BWS minimizes response fatigue and cognitive burden on respondents by limiting the number of items to be assessed to a few in respondent‐friendly tasks (Chrzan & Peitz, 2019).…”

Assessing patient information needs for new antidiabetic medications to inform shared decision‐making: A best–worst scaling experiment in China

“… 22 BWS might outperform rating scales in terms of both the ability to predict preferences and test‐retest reliability. 23 , 24 …”

“…22 BWS might outperform rating scales in terms of both the ability to predict preferences and testretest reliability. 23,24 To address the evidence gap, we aimed to investigate the patient information needs of new antidiabetic medications using the BWS experiment. Our findings would inform clinicians to provide patients with the desired information, thus promoting the implementation of SDM.…”

“…A comparison of the BWS and rating scales found that the BWS approach helps to validate priorities from a respondent perspective by achieving better discrimination among attributes 22 . BWS might outperform rating scales in terms of both the ability to predict preferences and test‐retest reliability 23,24 …”