How to Avoid the Sins of Questionnaire Abridgement - guideline

Abstract

The creation of abridged versions of research tools is a common, justifiable process, but unfortunately it is often carried out without due methodological care and regard for the consequences. Smith and collaborators (2000) have already written about the mistakes that can be made, but their article has not had much practical impact. There are two main mistakes commonly made by researchers: assuming the transferability of validity and reliability between the full and shortened versions and using less stringent criteria to assess the validity and reliability of short forms. These two problems manifest as nine sins committed during the construction of short forms. Here we present procedures designed to allow researchers to avoid these mistakes and create abridged versions of research tools that are as reliable as possible and to assess the costs of the various methods of abridging questionnaires. To this end we determine the expected length of the tool and weight the benefits of reduced questionnaire completion time against the loss of reliability. We also estimate the shared variance of the full and short versions and classification accuracy of the new, short version. We compared quality of short form obtained from three most common statistical techniques for abridging questionnaires. We analysed data from a sample of 519 persons; 309 (59.5%) completed the paper version of the Self-Narrative Inclination Questionnaire (IAN-R), and 210 (40.5%) participated in online tests. Abridgements based on factor loadings and Cronbach’s # were similarly effective and these two methods had a slight advantage over a method based on item response theory-based analyses of difficulty and discriminatory power.

Exploratory Factor Analysis Trees: Evaluating Measurement Invariance Between Multiple Covariates

“…Second, EFA trees can assist researchers in shortening questionnaires or in item selection, more generally. In practice, one of the main drivers when selecting "good" from "bad" items is the magnitude of factor loadings (Kleka & Soroko, 2018). However, this neglects the fact that even items with a small loadings might be important from a content validity standpoint.…”

Scale Development via Network Analysis: A Comprehensive and Concise Measure of Openness to Experience

“…In traditional psychometric approaches, item selection is typically determined by item-scale correlations and factor analysis loadings in classical test theory (CTT), and by item difficulty, item discrimination, and item information parameters in item response theory (IRT; Kleka & Soroko, 2018). In psychometric network analysis, the most common measure of an item's relationship to the overall construct is node centrality, which measures the influence of each node based on its relative position to other nodes in the network.…”

“…This provides an advantage over more traditional approaches in that it allows researchers to evaluate the structure of the construct's hierarchy and determine how to emphasize certain facets of the construct (Kleka & Soroko, 2018). In addition, the measure closely resembles the closeness centrality index of node centrality, making the interpretation intuitive with measures that researchers using network analysis are already familiar with.…”

Which scale short form development method is better? A Comparison of ACO, TS, and SCOFA

“…The main reason for this situation is to reduce the time and cost required for the application of the test, and the effort and length of the test that the participant would spend on the test items are appropriate (Kleka & Soroko, 2018). Due to these important reasons, academic studies to shorten the long forms of the scales have started to gain an important place in the social science literature.…”