Though Forced, Still Valid: Psychometric Equivalence of Forced-Choice and Single-Statement Measures

Abstract

Forced choice (FC) measures are gaining popularity as an alternative assessment format to single statement (SS) measures due to their potential in reducing the impact of various response styles and faking. However, a fundamental question remains to be answered: do FC and SS instruments measure the same underlying constructs? In addition, FC measures are theorized to be more cognitively challenging, so how would this feature influence respondents' reactions to FC measures compared to SS? Two studies were designed to answer these questions. Study 1 results showed that FC measures scored by the Multi-unidimensional Pairwise Preference Model (MUPP) and SS measures scored with an ideal point model yielded similar factor structures and almost identical criterion-related validity across 12 criteria. Both formats also had similar pattern of marginal reliabilities and test-retest reliabilities. Study 1 findings were replicated in Study 2.In addition, we found strong evidence for convergent validity between the two formats. Though the FC format was perceived to be more difficult, respondents showed no differential preference and expressed similar level of emotional and cognitive reactions to the two formats.

Controlling for Response Biases in Self-Report Scales: Forced-Choice vs. Psychometric Modeling of Likert Items

“…The empirical reliabilities presented in the main diagonal of Table 5 were acceptable for the GSQ and somewhat lower for FCQ, which is consistent with Zhang et al (2019). This may occur because, given their dichotomous format, pairwise forced-choice responses provide less information about the respondents’ trait levels than graded-scales.…”

Wetzel, Frick, & Brown (2020)_Does multidimensional forced-choice prevent faking?

“…A tutorial and Excel macro for creating the Mplus syntax can be found on http://annabrown.name/software. 2016; Zhang et al, 2019), while generally having lower reliabilities given the same items as their single-stimulus counterparts (Brown & Maydeu-Olivares, 2018b).…”

Comparing the validity of trait estimates from the multidimensional forced-choice format and the rating scale format.

“…We only summarize studies that applied Thurstonian item response modeling in the following because this is the model we also applied. For an investigation of validity-related issues in the context of an instrument with ideal point items using the multi-unidimensional pairwise preference model seeZhang et al (2019).…”