Integrative Multi-Modal Computing for Personal Health Navigation

An individual’s health trajectory is most influenced by personal lifestyle choices made regularly and frequently. It is now possible to measure, store, and analyze both multimodal lifestyle signals as well as multimodal physiological and behavioral health signals perpetually. Moreover, it is possible to model the effects of lifestyle precisely on health signals by collecting a variety of longitudinal data. These actions provide the inputs that affect changes in the health state of an individual based on their lifestyle decisions. With the advent of modern relatively inexpensive and common multi-modal data streams and sensing technologies, individuals now have large amounts of data and information about themselves that have the potential to transform decision-making in day-to-day life for health improvement. Multimodal analytics and contextual prediction and retrieval allow this data to make the best decisions to keep the health state optimal while making good lifestyle choices. This critical problem requires making a large variety of data constantly useful, contextually relevant, and most importantly useful in personalized decision-making. In order to address this challenge, we implement a generalized Personal Health Navigation (PHN) framework. PHN takes individuals toward their personal health goals through a system that perpetually digests multi-modal data streams from diverse sources, estimates current health status, computes the best route through intermediate states utilizing personal models, and guides the best inputs that carry a user towards their goal. We show the effectiveness of this approach using two examples in cardiac health. First, we prospectively test a knowledge-infused cardiovascular PHN system with a pilot prospective experiment of 41 users. Second, we create a data-driven personalized model on cardiovascular exercise response variability on a smartwatch data set of 33,269 real-world users. We conclude with critical challenges in multi-modal computing for PHN systems that require deep future investigation.