PROJECT SUMMARY
Listening effort and fatigue have serious impacts on quality of life for individuals with hearing loss. As the
clinical importance of listening effort becomes more widely acknowledged, there is an increased urgency to
improve our understanding of effort and fatigue in contexts relevant to real-world communication. One factor
that has the potential to alleviate effort and fatigue in everyday listening is the ability to see a conversation
partner’s face. While visual speech cues are known to improve speech intelligibility — particularly for listeners
with hearing loss — the link between visual speech and listening effort has not been established. Translating
the perceptual benefits of seeing a talker’s face to real listening further hinges on moving beyond video-based,
non-interactive contexts and investigating how listeners use visual speech cues during social communication.
The goals of this project are twofold. First, we will determine how and when seeing a talker’s face alleviates
effort and fatigue for listeners with cochlear implants (CIs), focusing on cognitively and perceptually demanding
listening scenarios. Second, we will examine eye gaze behavior during live conversations: how social
interaction changes where on the face listeners tend to look, how these patterns differ for listeners with CIs
(who likely rely more on visual cues from the talker’s mouth), and how individual differences in gaze behavior
relate to the effort of speech perception during conversation. Results will improve our understanding of
audiovisual speech perception and effort during natural communication, laying the groundwork for future
studies on how listeners with CIs use visual speech to improve the perception of prosody and track multi-talker
conversations. This work will inform the development of diagnostic testing to assess multisensory speech
perception, as well as rehabilitation strategies to maximize audiovisual benefits for individuals with hearing
and/or vision loss.