Project Summary
Neuropsychiatric disorders are the single greatest cause of disability due to non-communicable disease
worldwide, accounting for 14% of the global burden of disease. The current standards of care suffer from
subjectivity, inconsistent delivery, and limited access with growing waitlists. Data science solutions, in particular
artificial intelligence (AI) that can port to more ubiquitous mobile tools and that are not restricted for use in clinical
settings, have great potential to complement or even replace aspects of the standards of care. We propose to
develop a novel data science solution for one of the most pressing mental health burdens, autism, which is up
in prevalence by more than 200% since 1990, among the fastest growing pediatric concerns today, and highly
representative of many other mental health conditions. We have invented a prototype mobile system called
Guess What (GW) that noninvasively turns the focus of the camera on the child through a fluid social
engagement with his/her social partner in a way that reinforces prosocial learning while simultaneously
measuring the child’s developmental learning progress. At its simplest level, the GW app engages and
challenges the child to imitate social and emotion-centric prompts shown on the screen of a smartphone held
just above the eyes of the individual with whom the child is playing. Preliminary work to-date resulted in positive
user feedback, evidence of high engagement for both the parents and children, and meaningful gains in
socialization in the child. A single session produces 90 seconds of enriched social video and sensor data,
opening up an exciting opportunity for the game play to passively generate labeled training libraries that enable
the development of novel models that are extremely difficult to build without sufficient amounts of domain-
relevant training data. Our grant plan will explore this opportunity by designing and optimizing game modes,
creating a reusable active learning framework for growth of domain-relevant training libraries, and by creating at
least 3 “autism-feature-aware” neural networks that detect child emotion, eye gaze, and hand gestures. Our
project will show that GW can not only gamify crowdsourced construction of novel AI models that automatically
classify important features of child development – providing a way to address many challenges with AI in
medicine today -- but that it can also serve as a mobile therapy for repeat use to target core autism deficits while
also tracking progress at the same time.