“Use the Force!” Adaptation of response formats: From rating scale to multidimensional forced choice

Abstract

The present paper features the adaptation of an existing Big Five questionnaire with a rating scale (RS) response format into a measure using a multidimensional forced choice (MFC) response format. Rating scale response formats have been criticized for their proneness to intentional and unintentional response distortions. Multidimensional forced choice response formats were suggested as a solution to mitigate several types of response sets and response styles by design. The “Big Five Inventory of Personality in Occupational Situations” (B5PS) is a situation-based questionnaire designed for personnel selection and development purposes which would benefit from “fake-proof” response formats. MFC response formats require special effort during test construction and calibration which will be laid out here. Changing the response format has severe consequences on item design and scoring. An inherent issue with MFC formats derives from their inability to yield interpersonal comparative results from standard (sum) scoring. This issue can be solved with IRT based calibration (TIRT) during test construction. Aspects of MFC item design and TIRT calibrations are explored in this paper and evidence on structural and construct validity are presented. Results support the feasibility of the concept and test construction process in a contextualized/situation-based item format.