SCH: An Augmented Reality Neurorehabilitation System for Monitoring and Management of Motor Symptoms of Parkinson's Disease - PROJECT SUMMARY (See instructions): This project proposes augmented reality eyeglasses that can be worn throughout the day by a person with Parkinson's disease (PD) undergoing neurorehabilitation. The system enables (1) monitoring of and feedback to the patient during clinic visits, with provider-supervised physical therapy (PT) exercises, (2) display of immersive guidance and feedback to the patient during PT exercises at home that reference exercises supervised in the clinic, and (3) monitoring, analysis, and summarization of the patient’s activities of daily life (ADLs) between clinic visits, through cameras integrated into the eyeglasses and inertial sensors in shoes and wristbands. To develop such a system, we will advance the state-of-the-art in four research thrusts - egocentric tracking and reconstruction, egocentric visual perception, personalized reasoning models, and immersive analytics. In the first thrust, we will investigate the use of deep neural network architectures for highly accurate, clinically useful egocentric tracking and 3D reconstruction, enabling us to meaningfully record, playback, and evaluate rehabilitation activities. In the second thrust, we will develop a rich semantic understanding of contextualized activities using multimodal deep learning models that take into account short-term and long-term data from wearable inertial sensors and multiple egocentric cameras. In the third thrust, we will develop novel reasoning systems based on graphical models that leverage techniques such as partially observable Markov decision processes (POMDPs) to track and predict aspects of user performance during rehabilitation activities. In the fourth thrust, we will design an immersive analytics front-end interface that will integrate with our reasoning systems to provide customized user feedback and clinician summaries, enabling improved patient care. Developments across each of these thrusts will be integrated into an eyeglasses-based augmented reality system that will be iteratively evaluated and refined, first with able-bodied participants in a controlled lab setting, next with Parkinson’s patients in a clinical setting, then with able-bodied participants in home settings, and finally with Parkinson’s patients in their home environment.
