Estimating the reliability of emotion measures over very short intervals: The utility of within-session retest correlations.

Abstract

Short measures are commonly used when conducting research involving emotions. However, obtaining appropriate estimates of reliability for short measures is traditionally problematic and is a reoccurring concern in emotion research. To address this issue, we compare the withinsession test-retest and factor analysis methods for estimating the reliability of items in the PANAS-X. Results indicate that within-session test-retest ( ( ) ) estimates outperform the factor analysis method by demonstrating stronger relationships with item properties relevant to reliability and validity-related criteria. In addition, ( ) estimates appropriately generalize across samples with various instruction stems and prevent corrections for attenuation greater than 1.00. Therefore, we encourage researchers to use the corresponding average item-level ( ) estimates reported here to correct for attenuation when examining single-items from the PANAS-X if a test-retest design is not feasible.

Adjusting Bilingual Ratings by Retest Reliability Improves Estimation of Translation Quality

“…We can interpret this situation as meaning that we would obtain the same ordering of expected scores on the measures—that is, the scores participants would receive on the test in each language if averaging their responses across a large (conceptually infinite) number of repeated assessments—although they do not provide a consistent ordering of scores across single assessments . Although within-session retest correlations may serve as particularly useful reliability estimates of psychological states (Lowman et al, 2018), it seems likely that low within-session retest correlations may often indicate that participants have interpreted a specific item with reference to their current state, which may fluctuate considerably even within a 15-min retest interval. For instance, the .39 within-session retest correlation for the item “tired, exhausted” may come from participants interpreting the item as a state (how tired I am right now ) rather than as a trait (how tired I tend to be generally ).…”

“…The ability to collect all necessary data from participants who have only completed a single survey session reduces the experimenter and participant resources necessary to complete the study, which should make it easier to obtain larger sample sizes. In addition, as correlations tend to decrease in magnitude as the scores being correlated are separated farther in time, collecting the measures necessary to adjust for score unreliability within a single session has the expected effect of increasing interitem correlations (Lowman et al, 2018). Both of these features will serve to result in more stable estimates of the translation quality.…”

“…The repetition of the instrument thus is separated by a mere 10 min in which participants rate other measures. As argued by Lowman and colleagues (2018) and Wood and colleagues (2018), this method of estimating within-session retest correlations provides feasible reliability estimates for operationalizing Equation 1 because retesting even over an interval of 10 min (in which participants rate many other items) should be sufficient to largely eliminate participants’ memory of the specific answers they have given previously. Other properties of modern online surveys—such as the ability to easily randomize the order of measures and items and to prevent the possibility of looking back to one’s previous answers—should further increase the independence of within-session repeated measurements.…”

“…For instance, McCrae, Kurtz, Yamagata, and Terracciano (2011) found retest correlations to better track scale validity coefficients than internal consistency statistics such as alpha. As shown by de Vries, Realo, and Allik (2016) and Lowman, Wood, Armstrong, Harms, and Watson (2018), retest correlations also help to resolve the vexing problem of how to estimate the reliability of single items, as scales of any length can be retested. But perhaps even more importantly, these studies have found retest correlations to better track item-level validity coefficients (e.g., self–other agreement, long-term stability) than other coefficients which fall in the family of internal consistency coefficients (e.g., the squared communality of the item within a multiitem scale; Denissen, Geenen, Selfhout, & van Aken, 2008; Wanous & Hudy, 2001).…”

Descriptive, Predictive and Explanatory Personality Research: Different Goals, Different Approaches, but a Shared Need to Move beyond the Big Few Traits

“…Some may think that item-level findings are notoriously unreliable. But as was discussed before, items often have retest reliabilities of .65 or higher (Lowman, Wood, Armstrong, Harms, & Watson, 2018;Wood, Nye, & Saucier, 2010;Henry & Mõttus, 2020), which may be higher than many intuitively expect. Higher-than-assumed single item reliability is also consistent with findings that items out-predict scales for outcomes and other variables (Achaa-Amankwaa, Olaru, & Schroeders, 2020;Elleman, McDougald, Condon, & Revelle, 2020;Mõttus & Rozgonjuk, 2019;Seeboth & Mõttus, 2018;Vainik, Mõttus, Allik, Esko, & Realo, 2015).…”