Shortening of psychological tests – assumptions, methods and doubts

Kleka, Paweł; Paluchowski, Władysław Jacek

doi:10.1515/ppb-2017-0058

Cited by 5 publications

(7 citation statements)

References 22 publications

(17 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This brings the short form of L-POST in the range of preferred testing time of 10-15 minutes (Vancleef et al, 2019). Psychometric properties of any short form of L-POST will need to be evaluated, before it can be of clinical value in the diagnosis of perceptual organisation deficits after brain injury (Kleka & Paluchowski, 2017;Kruyen et al, 2013;Smith et al, 2000). Estimates of reliability and validity as reported in the Introduction are often overestimating empirical values and should only be used to guide development (Nunnally & Bernstein, 1994;Smith et al, 2000).…”

Section: Discussionmentioning

confidence: 99%

Shortening the Leuven Perceptual Organisation Screening Test with Item Response Theory and Confirmatory Factor Analysis

Vancleef¹,

Demeyere²,

Sun³

2021

Preprint

View full text Add to dashboard Cite

The Leuven Perceptual Organisation Screening Test (L-POST) is a reliable and validated test for mid-level visual perceptual deficits after brain injury. However, the test’s duration (20-35 minutes) is too long for a screening approach for all patients in clinic practice. Our aim was to shorten L-POST to 10-15 minutes based on statistical criteria of the items. We collected a sample of 3391 participants who completed L-POST. The test consists of 15 subtests with 5 items each. First, we demonstrated high internal consistency of the subtest items through Cronbach’s alpha and observed high correlations between scores based on all five items versus a selection of only two items per subtest. This showed that two items per subtest are sufficient. Next, Item Response Theory (IRT) was applied to guide the selection of the items. The highest correlation with full-scale subtest scores was observed when two items were selected following an adaptive testing procedure. A pilot validation in a subsample of participants with low abilities demonstrated adaptive testing has reasonable sensitivity (79%) but limited specificity (54%) in classifying participants with impaired and unimpaired abilities. Last, we reduced the number of subtests through factor analysis. We showed that the subtests using Radial Frequency Patterns as stimuli were redundant and could be combined in one subtest. We conclude that L-POST can be shortened to 26 items (7-14 minutes) and when used with fixed items could prove particularly useful for screening purposes.

show abstract

Section: Discussionmentioning

confidence: 99%

Shortening the Leuven Perceptual Organisation Screening Test with Item Response Theory and Confirmatory Factor Analysis

Vancleef¹,

Demeyere²,

Sun³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Additional data on L-POST in a larger sample of people with brain injury will allow us to investigate this crucial DIF for brain injury in the future. Indeed, psychometric properties of any short form of L-POST will need to be evaluated before it can be of clinical value in the diagnosis of perceptual organization deficits after brain injury (Kleka & Paluchowski, 2017; Kruyen et al, 2013; Smith et al, 2000).…”

Section: Discussionmentioning

confidence: 99%

Shortening the Leuven Perceptual Organization Screening Test with item response theory and confirmatory factor analysis.

Vancleef¹,

Demeyere²,

Sun³

2021

Psychological Assessment

View full text Add to dashboard Cite

The Leuven Perceptual Organization Screening Test (L-POST) is a test for midlevel visual perceptual deficits after brain injury. Previous research demonstrated reliable test scores and valid test score interpretations. However, the test's duration (20-35 min) is too long for a screening approach for all patients in clinical practice. Our aim was to shorten L-POST to 10-15 min based on statistical criteria of the items. Data from 3,391 participants who completed L-POST were analyzed. L-POST consists of 15 subtests with five items each. Item Response Theory (IRT) was applied to guide the selection of the items. The highest correlation with full-scale subtest scores was observed when two items were selected for each subtest following an adaptive testing procedure. A pilot validation in a subsample of participants with low abilities demonstrated adaptive testing has reasonable sensitivity (80%) but limited specificity (55%) in classifying participants with impaired and unimpaired abilities. Subsequently, we reduced the number of subtests through factor analysis. We showed that the subtests using Radial Frequency Patterns as stimuli were redundant and could be combined in one subtest. We conclude that L-POST can be shortened to 26 items (7-14 min) and an adaptive item selection procedure could prove particularly useful for screening purposes. Public Significance StatementTests for visual problems after brain injury are time-consuming and therefore not often used. In the current research study, we have used sophisticated statistical analyses to shorten an existing test in the best possible way. A shorter test can help remit underdiagnoses and enable access to appropriate treatment.

show abstract

“…Our analyses suggest that SFs produced using different statistical techniques yield rather similar scores in practice (at least with regard to the constructs analysed so far; cf. Kleka, 2013;Kleka & Paluchowski, 2017), but the most advanced techniques, which apply the most stringent assumptions, i.e. IRT models, offer greatest control over how the abridgement affects the psychological meaning of scores on an instrument.…”

Section: Discussionmentioning

confidence: 99%

“…It seems that the final score is more dependent on the content of items than their statistical properties (cf. Kleka, 2013;Kleka & Paluchowski, 2017). We carried out analyses not reported in this article, examining constructs from various psychological domains (intelligence; temperament; workaholism; self-narrative inclination) that are measured with multi-dimensional tools based on Likert scales.…”

Section: Limitations and Directions For Further Researchmentioning

confidence: 99%

How to Avoid the Sins of Questionnaire Abridgement - guideline

Kleka¹,

Soroko²

2018

Preprint

Self Cite

View full text Add to dashboard Cite

The creation of abridged versions of research tools is a common, justifiable process, but unfortunately it is often carried out without due methodological care and regard for the consequences. Smith and collaborators (2000) have already written about the mistakes that can be made, but their article has not had much practical impact. There are two main mistakes commonly made by researchers: assuming the transferability of validity and reliability between the full and shortened versions and using less stringent criteria to assess the validity and reliability of short forms. These two problems manifest as nine sins committed during the construction of short forms. Here we present procedures designed to allow researchers to avoid these mistakes and create abridged versions of research tools that are as reliable as possible and to assess the costs of the various methods of abridging questionnaires. To this end we determine the expected length of the tool and weight the benefits of reduced questionnaire completion time against the loss of reliability. We also estimate the shared variance of the full and short versions and classification accuracy of the new, short version. We compared quality of short form obtained from three most common statistical techniques for abridging questionnaires. We analysed data from a sample of 519 persons; 309 (59.5%) completed the paper version of the Self-Narrative Inclination Questionnaire (IAN-R), and 210 (40.5%) participated in online tests. Abridgements based on factor loadings and Cronbach’s # were similarly effective and these two methods had a slight advantage over a method based on item response theory-based analyses of difficulty and discriminatory power.

show abstract

Shortening of psychological tests – assumptions, methods and doubts

Cited by 5 publications

References 22 publications

Shortening the Leuven Perceptual Organisation Screening Test with Item Response Theory and Confirmatory Factor Analysis

Shortening the Leuven Perceptual Organisation Screening Test with Item Response Theory and Confirmatory Factor Analysis

Shortening the Leuven Perceptual Organization Screening Test with item response theory and confirmatory factor analysis.

How to Avoid the Sins of Questionnaire Abridgement - guideline

Contact Info

Product

Resources

About