Cross-Modal Health State Estimation

Abstract

Individuals create and consume more diverse data about themselves today than any time in history. Sources of this data include wearable devices, images, social media, geo-spatial information and more. A tremendous opportunity rests within cross-modal data analysis that leverages existing domain knowledge methods to understand and guide human health. Especially in chronic diseases, current medical practice uses a combination of sparse hospital based biological metrics (blood tests, expensive imaging, etc.) to understand the evolving health status of an individual. Future health systems must integrate data created at the individual level to better understand health status perpetually, especially in a cybernetic framework. In this work we fuse multiple user created and open source data streams along with established biomedical domain knowledge to give two types of quantitative state estimates of cardiovascular health. First, we use wearable devices to calculate cardiorespiratory fitness (CRF), a known quantitative leading predictor of heart disease which is not routinely collected in clinical settings. Second, we estimate inherent genetic traits, living environmental risks, circadian rhythm, and biological metrics from a diverse dataset. Our experimental results on 24 subjects demonstrate how multi-modal data can provide personalized health insight. Understanding the dynamic nature of health status will pave the way for better health based recommendation engines, better clinical decision making and positive lifestyle changes.

Food Recommendation: Framework, Existing Solutions, and Challenges

“…For example, Nag et al [9] combined food images and GPS context for foodlog. They [45] then fused multiple user source data streams along with the domain knowledge for crossmodal health state estimation. Recently, Markus et al [46] used different kinds of features from different modalities, including a recipe's title, ingredient list and cooking directions, popularity indicators (e.g., the number of ratings) and visual features to estimate the healthiness of recipes for recipe recommendation.…”

Flavour Enhanced Food Recommendation

“…Unfortunately these metrics fail to capture the quality of recommendations in relationship to real world implementation for enjoyment or health. E ectively extending the food recommendation to incorporate the individual health state criteria and culinary avour and user preferences will be the next evolution of more personalized food recommendation [4,6,7,10,12].…”

Ubiquitous Event Mining to Enhance Personal Health

“…These event associations form the hypotheses for the do-operator. An example hypothesis is Bike ω [ 2 , 4 ] W ork , which means that a bike event is followed by a work event within 2 to 4 hours.…”

“…Similar approaches can be used for ubiquitous computing and designing effective user interventions through smartphones, wearables, and IoT devices. We are constantly monitoring our life using a variety of devices and applications allowing us to estimate the state of different aspects of our life, health, and habits [4, 5]. We can take further advantage of this ubiquitous computing paradigm to develop a navigational approach to health, in which our computing services cater to our individual preferences and help actuate actions that lead us towards our health goals.…”