While Mobile Health research and use cases continue to grow, privacy protection continues to be an important implementation challenge. Using data from the University of Michigan Intern Health Study (IHS), we first construct a participant mood prediction model. We then perform a cost-benefit analysis of two privacy protection technologies (Federated Learning and Differential Privacy) as applied to this model. We find that Federated Learning alone does not provide adequate protection against simulated privacy attacks proposed in the literature, with participant risk of private data leakage over 90%. However, adding a sufficient amount of Differential Privacy reduces the attacker's success rate to 59.6% with only a 10 percentage point decrease in model R2 and a 42% increase in training time. Finally, we show that those participants in the IHS with more sensitive personal data are also most at risk from this particular privacy attack and subsequently stand to benefit the most from these privacy-preserving technologies.

Background Although evidence supporting the feasibility of large-scale mobile health (mHealth) systems continues to grow, privacy protection remains an important implementation challenge. The potential scale of publicly available mHealth applications and the sensitive nature of the data involved will inevitably attract unwanted attention from adversarial actors seeking to compromise user privacy. Although privacy-preserving technologies such as federated learning (FL) and differential privacy (DP) offer strong theoretical guarantees, it is not clear how such technologies actually perform under real-world conditions. Objective Using data from the University of Michigan Intern Health Study (IHS), we assessed the privacy protection capabilities of FL and DP against the trade-offs in the associated model’s accuracy and training time. Using a simulated external attack on a target mHealth system, we aimed to measure the effectiveness of such an attack under various levels of privacy protection on the target system and measure the costs to the target system’s performance associated with the chosen levels of privacy protection. Methods A neural network classifier that attempts to predict IHS participant daily mood ecological momentary assessment score from sensor data served as our target system. An external attacker attempted to identify participants whose average mood ecological momentary assessment score is lower than the global average. The attack followed techniques in the literature, given the relevant assumptions about the abilities of the attacker. For measuring attack effectiveness, we collected attack success metrics (area under the curve [AUC], positive predictive value, and sensitivity), and for measuring privacy costs, we calculated the target model training time and measured the model utility metrics. Both sets of metrics are reported under varying degrees of privacy protection on the target. Results We found that FL alone does not provide adequate protection against the privacy attack proposed above, where the attacker’s AUC in determining which participants exhibit lower than average mood is over 0.90 in the worst-case scenario. However, under the highest level of DP tested in this study, the attacker’s AUC fell to approximately 0.59 with only a 10% point decrease in the target’s R2 and a 43% increase in model training time. Attack positive predictive value and sensitivity followed similar trends. Finally, we showed that participants in the IHS most likely to require strong privacy protection are also most at risk from this particular privacy attack and subsequently stand to benefit the most from these privacy-preserving technologies. Conclusions Our results demonstrated both the necessity of proactive privacy protection research and the feasibility of the current FL and DP methods implemented in a real mHealth scenario. Our simulation methods characterized the privacy-utility trade-off in our mHealth setup using highly interpretable metrics, providing a framework for future research into privacy-preserving technologies in data-driven health and medical applications.

BACKGROUND While evidence supporting the feasibility of large scale mHealth systems continues to grow, privacy protection continues to be an important implementation challenge. The potential scale of publicly available mHealth applications and the sensitive nature of the data involved will inevitably attract unwanted attention from adversarial actors seeking to compromise user privacy. Although privacy-preserving technologies such as Federated Learning and Differential Privacy offers strong theoretical guarantees, it is not clear how such technologies actually perform under real-world conditions. OBJECTIVE Using data from the University of Michigan Intern Health Study (IHS), we assess the privacy protection capabilities of Federated Learning and Differential Privacy against the associated tradeoffs in model accuracy and training time using simulation methods. Specifically, our objectives are to (1) construct a “target” mHealth system using the demographic and sensor data available in the IHS (2) mount a simulated privacy attack that attempts to compromise IHS participant privacy (3) measure the effectiveness of such an attack under various levels of privacy protection on the target mHealth system, and (4) measure the costs to algorithmic performance associated with the chosen levels of privacy protection. METHODS For (1), we perform simple data processing/imputation and construct a neural network classifier that attempts to predict participant daily mood EMA score from sensor data. For (2) we make certain assumptions of the attacker’s capabilities and construct an attack intended to uncover statistical properties of private participant data based on techniques proposed in the literature. For (3) and (4), we report a collection of conventional metrics to evaluate the success of the privacy attack and performance of the original mHealth system under Federated Learning and various levels of Differential Privacy. RESULTS We find that Federated Learning alone does not provide adequate protection against the privacy attack proposed above, where the attacker’s success rate in identifying private data attributes is over 90% in the worst case. However, under the highest level of Differential Privacy tested in this paper, the attacker’s success rate falls to around 59.6% with only a 10 percentage point decrease in model R2 and a 42% increase in model training time. Finally, we show that those participants in the IHS most likely to require strong privacy protection are also most at risk from this particular privacy attack and subsequently stand to benefit the most from these privacy-preserving technologies. CONCLUSIONS Our results demonstrate both the necessity of proactive privacy protection research and the feasibility of current Federated Learning and Differential Privacy methods implemented in a real mHealth scenario. Our simulation methods for privacy protection measurement provide a novel framework for characterizing the privacy-utility tradeoff and serve as a potential foundation for future research.

No abstract