Information Theory, Machine Learning, and Bayesian Networks in the Analysis of Dichotomous and Likert Responses for Questionnaire Psychometric Validation

Abstract

The validation of questionnaires, crucial for discriminating between di-verse populations, is a standard practice in psychology and medicine. Whilelatent factor models have conventionally dominated psychometric question-naire validation, recent developments have introduced alternative method-ologies such as Network Analysis. This study presents a pioneering approachthat integrates information theory, machine learning (ML), and Bayesian net-works (BNs) into questionnaire validation. This novel perspective shifts theemphasis from latent factors to individual items. We used the Jensen-ShannonDivergence (JSDd) for item selection, employing three machine learning algo-rithms (Decision Trees, Random Forests, and Support Vector Machines witha linear kernel) to identify the items with optimal discriminative power. Theselection process balanced the number of items against model accuracy in adata-driven manner. Bayesian Networks (BNs) were employed to uncoverconditional dependences between items, offering insights into the complexsystems underlying the psychological construct.We validated the proposed method on two simulated data sets, one withdichotomous and the other with Likert-scale data. Results show the efficacy ofthe proposed method in identifying the most discriminative items, thereby en-hancing the instrument’s discriminative power. Simultaneously, it mitigatedrespondent burden by minimising the required number of administered itemsand providing insights into the criterion validity, content validity, and con-struct validity of the instrument.