Shifting gears: consequences of including two negatively worded items in the middle of a positively worded questionnaire

Abstract

A questionnaire used in student evaluations of interdisciplinary courses during six semesters contained two Likert items stated in a direct negative mode which were embedded in a questionnaire (14-18 items) in which the remaining items were phrased in a direct positive mode. In the seventh semester and thereafter, the two negative items were restated as direct positive stems. Item-analysis demonstrated that in the direct negative mode, the two items had low item-to-total correlations and that the internal consistency reliability of the sum score could be improved by eliminating the two negatively phrased items. Also, the two negatively worded items defined a separate factor. After they were reworded into a direct positive mode, these two items showed markedly improved item-to-total correlations. Moreover, the unique factor disappeared, which suggests that it was a methodological artefact probably attributable to respondent carelessness. Including a few negative items in an otherwise positively stated questionnaire leads to ambiguity of results rather than controlling for response sets. We therefore recommend against the practice.

Reprodutibilidade, consistência interna e validade de construto do KIDSCREEN-27 em adolescentes brasileiros

“…Adolescentes jovens têm dificuldades para entender itens negativos em escalas de medida 35 , e o seu uso pode resultar em menores níveis de precisão 36,37 .…”

TIMSS data in an African comparative perspective: Investigating the factors influencing achievement in mathematics and their psychometric properties

“…Studies have indicated that the method effect can be seen if at least about 10 % of respondents fail to acknowledge the presence of negatively worded items in a survey instrument and respond appropriately (Woods 2006;Schmitt and Stults 1985). Moreover, many studies on constructs validation (e.g., even for older students and adults) have shown that positively and negatively worded items in a survey often form two distinct factors structures in factor analysis (e.g., Marsh 1986Marsh , 1996Metsämuu-ronen 2012a;Roszkowski and Soven 2010;Chiu 2008Chiu , 2012. However, the split in factors is very much associated with uni-dimensional construct.…”

“…When method effects are not accounted for, they have been shown to affect the goodness of fit statistics, lead to biased parameter estimates (e.g., Chiu 2012Chiu , 2008Bagozzi 1993;Quilty et al 2009;Marsh 1994Marsh , 1996Distefano and Motl 2009;Marsh et al 2013;Lindwall et al 2012;Tomǭs et al 2013;Horan et al 2003;Magazine et al 1996) and influence the validity and reliability of the scale (e.g., Roszkowski and Soven 2010;Woods 2006;Raykov 2001;Brown 2015). Moreover, the method effect can lead to wrong inferences by suppressing or inflating the relationships between constructs by contributing to Type I or Type II errors if not incorporated into the measurement model (e.g., Bagozzi 1993; Magazine et al 1996).…”

Untitled

“…These reasons include (a) MacDonald (2000b) did not find a higher order EWB factor in his original study, (b) there have been questions raised regarding the appropriateness of incorporating any well-being concepts within measures of spirituality (including the ESI-R) as it results in a confounding of the two constructs which undermines the usability of spirituality measures in health research (Koenig, 2008;Migdal & MacDonald, 2013), and (c) other than one PAR item, the EWB items are the only ones on the ESI-R which are reverse scored and use more negative phrasing. Not only is it more challenging to effectively translate negatively worded items from one language to another, research has suggested that the inclusion of such items on a test which otherwise uses positively phrased items runs the risk compromising the quality of measurement (e.g., Roszkowski & Soven, 2010). Moreover, it is been found that younger testtakers tend to have difficulties with negatively worded items (e.g., Benson & Hocevar, 1985).…”