Influence of Response-Option Combinations when Measuring Sense of Efficacy for Teaching: Trivial, or Substantial and Substantive?

Abstract

Certain combinations of number and labeling of response options on Likert scales might, because of their interaction, influence psychometric outcomes. In order to explore this possibility with an experimental design, two versions of a scale for assessing sense of efficacy for teaching (SET) were administered to preservice teachers. One version had seven response options with labels at odd-numbered points; the other had nine response options with labels only at the extremes. Before outliers in the data were adjusted, the first version produced a range of more desirable psychometric outcomes but poorer test–retest reliability. After outliers were addressed, the second version had more undesirable attributes than before, and its previously high test–retest reliability dropped to poor. These results are discussed in relation to the design of scales for assessing SET and other constructs as well as in relation to the need for researchers to examine their data carefully, consider the need to address outlying data, and conduct analyses appropriately and transparently.

Measuring teacher self-efficacy: Validating a new comprehensive scale among Chinese pre-service teachers

“…The existence of two factors, especially the Ethos factor, is supported by the low proportion of missing data and infrequent use of the no idea and not applicable response options, suggesting the high relevance of the 40 purposely selected items as revealed in the preliminary relevance rating. The distinct factorial structure could also have benefited by applying the response scale with labels on odd numbers, which was reported to produce better data quality and psychometric results by assisting PSTs to rate TSE more accurately ( Trevethan and Ma, 2021 ). Meanwhile, this structure was supported because, first, two distinct factors were indicated in the parallel analysis; second, there was a totally distinct pattern of loadings on each domain in the EFA; third, the two-factor model fit in the CFA had a high degree of acceptability, and fourth, each factor comprised a meaningful cluster of items that, as noted above, corresponded with the two-factor solutions theorized and identified in other research (e.g., Friedman and Kass, 2002 ; Skaalvik and Skaalvik, 2007 ).…”

“…Participants were invited to indicate how effective you think you will be in each of the following activities with responses on the 9-point response scale with 1, 3, 5, 7, 8 and 9 being labeled minimally effective, only moderately effective, quite effective, extremely effective, I really have no idea at all and not applicable and 2, 4, and 6 not labelled. Such a method of labelling, as used by Skaalvik and Skaalvik (2007) outperformed only labelling the extreme responses in producing a higher quality of data and psychometric outcomes in our preliminary experimental research (Trevethan and Ma, 2021). In addition, the last two options were added due to PSTs' potential incapability with understanding certain items especially those without any teaching experience and the purpose to detect any potential items that might be irrelevant (see Duffin et al, 2012;Ma et al, 2020).…”