Abstract: Chronic conditions such as congenital heart disease, Alzheimer's disease and osteoarthritis affect a significant segment of the population. Today, an estimated 133 million Americans - nearly half the population - suffer from at least one chronic illness. Longitudinal studies that monitor subjects over extended periods of time help determine the relationships between risk factors and disease evolution, which is essential in quantifying the effectiveness of treatment and palliative care. The studies comprise multimodal data such as demographics, time series, medical images, and genetic information. All are collected across multiple institutions, multiple patient populations and multiple visits. The collection process induces heterogeneity at all levels: there is high irregularity, inter-subject variability, and potentially changing collection protocols. Reliable disease trajectory models, constructed through retrospective statistical analysis of this multimodal longitudinal data, are necessary to inform patients and facilitate clinical decisions.
We address the methodological gap by tightly integrating multimodal data and leveraging the different sources of information, including domain expertise, to extract salient features. In the Information Fusion Lab, we develop hybrid models that optimize multi-component objectives, specialized to the task and for the available data. Our models include hybrid layers, designed to cope with multiple inputs of distinct types, such as attributes encoded as discrete features provided together with their associated images. In the talk, I will present mechanisms to conditionally route samples through the neural networks depending on their cross-modal characteristics, models that leverage the intrinsic frequency of signals to learn sparse forecasting models from multivariate time series and weakly supervised deep learning architectures incorporating domain-specific heuristics. These techniques have enabled us to efficiently construct representations of images that adhere to specific patterns, such as medical images of different organs. In the talk, I will demonstrate the performance of our models in attaining state of the art results on tasks such as Alzheimer's disease forecasting, detecting heart conditions and in-hospital mortality prediction. Finally, I will describe how multimodal and multiresolution networks can be used for weather modeling.