“Use the Force!” Adaptation of Response Formats

Abstract

Abstract. The present paper features the adaptation of an existing Big Five questionnaire with a rating scale (RS) response format into a measure using a multidimensional forced choice (MFC) response format. Rating scale response formats have been criticized for their proneness to intentional and unintentional response distortions. Multidimensional forced choice response formats were suggested as a solution to mitigate several types of response sets and response styles by design. The Big Five Inventory of Personality in Occupational Situations (B5PS) is a situation-based questionnaire designed for personnel selection and development purposes which would benefit from fake-proof response formats. MFC response formats require special effort during test construction and calibration which will be laid out here. Changing the response format has severe consequences on item design and scoring. An inherent issue with MFC formats derives from their inability to yield interpersonal comparative results from standard (sum) scoring. This issue can be solved with item response theory (IRT)-based calibration during test construction. The Thurstonian IRT approach (TIRT) was developed by Brown and Maydeu-Olivares (2011) , and aspects of MFC item design and TIRT calibrations are explored in this paper. Evidence on structural and construct validity are presented alongside recommendations on the test development processes. The results support the feasibility of the concept of MFC test construction with TIRT calibration in a contextualized and situation-based item format.