Validating the Quality of Crowdsourced Psychometric Personality Test Items

Abstract

Personality questionnaire items are normally written by experts in personality theory or psychometrics. This makes personality test development expensive, especially since as many as 40% of the items are discarded after quantitative analysis. Could the crowd create personality items that are as good as the ones developed by professionals? We analyze the psychometric quality of 164 Big Five personality items generated by the crowd (n=82) and show that non-expert item writers are capable of designing high quality personality items. The analysis revealed that 59% of the crowdsourced items had adequate psychometric properties, and thus could be used as an alternative instrument to measure Big Five personality traits. We conclude that crowdsourcing personality items is a possible strategy to generate new items rapidly. We also discuss some of the challenges of crowdsourcing general psychological items.