Enhancing targeted outreach for longitudinal surveys: predictive analytics for participant response in the Millennium Cohort Study

Abstract

Background: The Millennium Cohort Study is a prospective cohort study designed to examine the long-term effects of military service. The study collects self-reported data from surveys administered every 3–5 years to military personnel and veterans. Participant nonresponse to follow-up surveys presents a potential threat to the validity and generalizability of study findings. In recent years, predictive analytics has emerged as a promising tool to identify predictors of nonresponse. Methods: Here, we present a method that leverages machine learning techniques to develop a high-skill classifier to predict participant response to Millennium Cohort Study follow-up surveys. Using a temporal cross-validation method, six supervised algorithms, each using differing learning strategies, were employed to predict response to the 2021 follow-up survey. Using latent class analysis (LCA), we classified participants based on historical survey response and compared prediction performance with and without this variable. Feature analysis was subsequently conducted on the best performing model to identify the most important factors. Results: The LCA variable consisted of six distinct classes: consistent web responders, consistent paper responders, mode switchers, early and late consistent nonresponders, and inconsistent responders. Notably, the LCA classes were found to be statistically significant across various military and demographic characteristics. Including the LCA variable in the machine learning analysis, all six algorithms performed comparably to one another. However, without the LCA variable, random forest outperformed the benchmark regression model by 9.3% in area under the curve (AUC) of the receiver operating characteristic (ROC) curve and 7.7% in AUC of precision-recall. Moreover, inclusion of the LCA variable increased the ROC AUC for all algorithms by 10% or more, except for random forest and interaction forest, which improved by approximately 5% in ROC AUC. Feature analysis indicated LCA was the most important predictor in the model. Conclusions: Our findings highlight the importance of historical response patterns to significantly improve prediction performance of participant response to follow-up surveys. Machine learning algorithms can be especially valuable when historical data are not available. Implementing these methods in longitudinal studies can enhance outreach efforts by strategically targeting participants, ultimately boosting survey response rates and mitigating nonresponse.