A multilevel structural equation model for assessing a drug effect on a patient‐reported outcome measure in on‐demand medication data

Abstract

We analyze data from a clinical trial investigating the effect of an on-demand drug for women with low sexual desire. These data consist of a varying number of measurements/events across patients of when the drug was taken, including data on a patient-reported outcome consisting of five items measuring an unobserved construct (latent variable). Traditionally, these data are aggregated prior to analysis by composing one sum score per event and averaging this sum score over all observed events. In this paper, we explain the drawbacks of this aggregating approach. One drawback is that these averages have different standard errors because the variance of the underlying events differs between patients and because the number of events per patient differs. Another drawback is the implicit assumption that all items have equal weight in relation to the latent variable being measured. We propose a multilevel structural equation model, treating the events (level 1) as nested observations within patients (level 2), as alternative analysis method to overcome these drawbacks. The model we apply includes a factor model measuring a latent variable at the level of the event and at the level of the patient. Then, in the same model, the latent variables are regressed on covariates to assess the drug effect. We discuss the inferences obtained about the efficacy of the on-demand drug using our proposed model. We further illustrate how to test for measurement invariance across grouping covariates and levels using the same model.

A comparison of the multilevel MIMIC model to the multilevel regression and mixed ANOVA model for the estimation and testing of a cross‐level interaction effect: A simulation study

“…Furthermore, when applying the ML‐MIMIC model in practice, it is important to verify the measurement invariance assumptions. In ML‐MIMIC models, invariance of factor loadings and intercepts across groups is implicitly assumed (as was the case in our simulations), but it is advised to test these strict assumptions (Kessels et al., 2021; Kim et al., 2015), in order to ensure that unbiased covariate effects are obtained. Research has shown that in situations where the measurement invariance assumption is partly violated (some but not all intercepts and/or factor loadings are invariant), covariate effects are still unbiased as long as the model is corrected for the invariant parameters (Hsiao & Lai, 2018).…”

“…The presence of response shift should therefore also be investigated in ML‐MIMIC models and it has been shown that response shift plays a role in studies using patient reported outcomes, such as quality of life outcomes in cancer studies (Ilie et al., 2019). However, how to deal with these possible measurement invariance violations is beyond the scope of this study and we refer for this to other work (Carlier et al., 2019; Kessels et al., 2021; Kim et al., 2015).…”

“…The mixed ANOVA model was applied to an aggregated data structure. Here, we follow the procedure similar to previous work on this topic (Kessels et al., 2019, 2021), which was motivated by the real‐world clinical trial example outlined in Section 1. This means that the univariate outcome $$Y_{ei}$$ defined in the previous section, which denotes the sum of the observed scores at event e for patient i , was averaged over all events belonging to the same level of the within‐level grouping covariate within a patient.…”

“…The potential issues with aggregating the sexual event data have led to the motivation of proposing the ML‐MIMIC model for analyzing such data (Kessels et al., 2021), because the ML‐MIMIC model does not require aggregated data. ML‐MIMIC models have been gaining popularity in educational and psychological research (Davidov et al., 2019; Jak et al., 2014; Roesch et al., 2010), but these models are less commonly used for analyzing patient reported outcomes in clinical trials.…”

Limitations of the Sum-and-Alpha Approach to Measurement in Behavioral Research

“…If the data are poor-as when there is no validity evidence supporting scores-the most sophisticated data analysis will remain susceptible to incorrect conclusions and inaccurate estimates (Flake, 2021). Weaknesses of unsubstantiated sum scoring have been demonstrated with advanced statistical approaches such as regression discontinuity (Soland et al, 2022), machine learning (Jacobucci & Grimm, 2020), intensive longitudinal data and time-series analysis (McNeish et al, 2021), growth modeling (Kuhfeld & Soland, 2022), network modeling (Haslbeck et al, 2022), and clinical trials (Kessels et al, 2021). More simply, the quality of the statistical model is limited by the quality of the data itself and no statistical workaround can avoid consequences of poor measurement.…”