PROJECT SUMMARY/ABSTRACT
Blunted or flat facial expression is characteristic of schizophrenia (Sz) spectrum disorders and their clinical high
risk (CHR) states, and is associated with negative symptoms, social impairment, and poor outcome. Currently,
there is no objective test to quantify blunted face expression in Sz. However, with advancements in computational
methods, we can begin to operationalize blunted face expression, necessary for both methodological studies
and clinical trials, especially as there are no evidence-based treatments. We propose computational analyses of
time series of video frame-based estimates of movements of individual face muscles, based on Ekman and
Friesen’s Facial Action Coding System or FACS, obtained during ecologically valid open-ended interview on a
remote platform (and during a standard viewing paradigm of valenced stimuli). Face action units (AUs) index
movement of individual face muscles, which have known physiology and circuitry. Our preliminary data are
consistent with early small video coding and electromyography studies of face AUs in Sz, in replicating decreased
mean amplitude of face AUs, especially of AU7 (“lid tightener”) or orbicularis oculi, which is involved in smiling,
but also social signaling more broadly. Further, in generating matrix and distance profiles for face AU time series
data, we find indices of decreased repertoire of face expression in Sz, also associated like amplitudes in CHR
with ratings of blunted affect and poor function. Aim 1 is to assess the correlates and psychometric properties of
these face expression amplitude and repertoire metrics in a large international cohort of early course Sz
spectrum, CHR and healthy individuals well-characterized for demographics, symptoms, cognition and function,
(and in whom effects of medications and chronicity may be less). These metrics are expected to differentiate
psychosis from the norm, have convergent validity with ground truth ratings of “blunted affect”, and be correlated
with social and role impairment, and also face processing ability. Variation by demographics (age, sex, ethnicity)
will be assessed, as well as test-retest reliability at one year. Aim 2 is to align video and audio time series to
create per frame sets of features and envelope metrics in segments in participants to test for differences in facial
dynamics when individuals are speaking vs. listening. We hypothesize that global face expression will decrease
significantly during pauses in individuals with Sz, consistent with preliminary data. Based on a cognitive model,
we hypothesize decreased face expression and pause behavior will be correlated and associated with slowed
processing speed. Aim 3 assesses synchrony of face expression between interviewers and participants, with
patients hypothesized to have decreased synchrony and alignment across modalities. Overall, this large rich
dataset of multimodal time series raw data will be archived and available for analyses, including more complex
nonlinear time series analyses.