Assessing intra- and inter-individual reliabilities in intensive longitudinal studies: A two-level random dynamic model-based approach.

Abstract

Intensive longitudinal studies are becoming increasingly popular because of their potential for studying the individual dynamics of psychological processes. However, measures used in such studies are quite susceptible to measurement error due to the short lengths and therefore their psychometric properties, such as reliability, are of great concern. Most existing approaches for assessing reliability are not appropriate for the intensive longitudinal data (ILD) because of the conflation of inter- and intra-individual variations or the difficulty in handling interindividual differences. In addition, measurement models are always relegated or omitted in the ILD modeling approaches. Therefore, in this article, we introduce a two-level random dynamic measurement (2RDM) model for ILD, which takes into account measurement models for key variables of interest. Then we discuss how to derive the within-person and between-person reliabilities for items and scales in the context of the 2RDM model. A small simulation study is presented to illustrate the implementation of the 2RDM model and reliability estimation. An empirical study is then provided to demonstrate the application of the proposed approach for multidimensional scales, in which we calculated the within- and between-person reliabilities for both items and subscales of a short version of the Perceived Stress Scale and found large individual differences in the within-person reliabilities. We conclude by discussing the advantages and considerations of the proposed approach in practice.

Developing, evaluating, and interpreting personality state measures: A framework based on the revised latent state-trait theory

“…For example, it has recently been suggested to estimate both between-and within-person reliabilities using a two-level random dynamic measurement model (Xiao, Wang, & Liu, 2023). Moreover, latent Markov factor analysis can be used to probe intrapersonal changes in measurement models over time or situations (Vogelsmeier, Vermunt, van Roekel, & De Roover, 2019).…”

The Many Reliabilities of Psychological Dynamics: An Overview of Statistical Approaches to Estimate the Internal Consistency Reliability of Intensive Longitudinal Data

“…To the best of our knowledge, these approaches include: (a) The generalizability theory (Cranford et al, 2006), (b) the multilevel framework (Nezlek, 2017), (c) the multilevel confirmatory factor analysis model (Geldhof et al, 2014;Lai, 2021), (d) the dynamic factor analysis approach (Fuller-Tyszkiewicz et al, 2017;Hu et al, 2016), (e) the latent state-trait theory (Castro-Alvarez et al, 2022), and (f) the DSEM framework (Xiao et al, 2023). Nevertheless, approaches that estimate the test-retest reliability of intensive longitudinal data have also been proposed for single-item measures, such as the multilevel (vector) autoregressive model with measurement error (Schuurman & Hamaker, 2019), and the test-retest reliability coefficients for experience sampling methods suggested by Dejonckheere et al (2022).…”

“…In our review of the literature, we identified three approaches to estimate the reliability of intensive longitudinal data that are fairly known and used by empirical researchers: The generalizability theory (Cranford et al, 2006), multilevel modeling (Nezlek, 2017), and multilevel confirmatory factor analysis (Geldhof et al, 2014;Lai, 2021). Additionally, we identified three novel approaches that have not been applied in empirical studies: The dynamic factor analysis approach (Fuller-Tyszkiewicz et al, 2017;Hu et al, 2016), the latent state-trait theory (Castro-Alvarez et al, 2022), and the two-level random dynamic model-based approach (Xiao et al, 2023). All of these approaches have in common that they allow for estimating the reliability of the scales used in intensive longitudinal research when several items are repeatedly administered to measure one construct (e.g., using the items energetic, happy, and enthusiastic to measure positive affect).…”

“…The last approach to estimate the reliability of intensive longitudinal data that we discuss in this article is the so-called two-level random dynamic model-based approach (2RDM; Xiao et al, 2023). This model has also been referred to as a dynamic factor vector autoregressive model (DFVAR; Oh & Jahng, 2022).…”

“…Then, the means and variances of the random coefficients (fixed effects) are also modeled at the between-level. As the fixed effects do not have a major role in the computation of the reliability coefficients, we suggest that the interested reader refers to the details of the modeling of these effects in Xiao et al (2023).…”