Project Summary
Neuropsychiatric disorders are the single greatest cause of disability due to non-communicable disease
worldwide, accounting for 14% of the global burden of disease. The current standards of care suffer from
subjectivity, inconsistent delivery, and limited access with growing waitlists. New informatics solutions, in
particular artificial intelligence (AI) that can port to more ubiquitous mobile health devices and that are not
restricted for use in clinical settings, have great potential to complement or even replace aspects of the standards
of care. We propose to develop a novel informatics solution for one of the most pressing mental health burdens,
autism, which is up in incidence by more than 600% since 1990, among the fastest growing pediatric concerns
today, and highly representative of many other neuropsychiatric conditions. We have invented a prototype
mobile system called Guess What (guesswhat.stanford.edu) (GW) that turns the focus of the camera on the
child through a fluid social engagement with his/her social partner that reinforces prosocial learning while
simultaneously measuring the child’s developmental learning progress. At its simplest level, the GW app
challenges the child to imitate social and emotion-centric prompts shown on the screen of a smartphone held
just above the eyes of the individual with whom the child is playing. But more, as a home-based repeat-use
system, GW uses computer vision algorithms and emotion classifiers integrated into gameplay to detect emotion
in the child’s face via the phone’s front camera, automatically finding agreement with the displayed prompt, while
capturing features such as gaze, eye contact, and joint attention. Preliminary work with more than 20 autistic
children resulted in positive user feedback, evidence of high engagement for both the parents and children, and
importantly, evidence of clinically meaningful gains in socialization. A single session produces 90 seconds of
enriched social video and sensor data, opening up an exciting opportunity for the game play itself to passively
generate labeled computer vision libraries that enable the development of better models with higher diagnostic
precision going forward. Our proposed project will show that GW can (a.) serve as a mobile therapy that can be
used repeatedly by families to target core deficits of autism while inherently tracking progress during use, and,
(b.) serve as a distributed system to crowdsource the acquisition of new labeled image libraries for AI models
that can automatically classify diagnostic features relevant to autism and extend to other sectors of mental health
(and even beyond).