Official websites use .gov

A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS

A lock ( 🔒 ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Need Help

Semantic-based auditory attention decoder using Large Language Model

Award Number: R16GM159795
ORGANIZATION: NATIONAL INSTITUTE OF GENERAL MEDICAL SCIENCES
OPDIV: NIH
AWARD CLASS: DISCRETIONARY
AWARD ACTIVITY TYPE: SCIENTIFIC/HEALTH RESEARCH (INCLUDES SURVEYS)
PERIOD OF PERFORMANCE START DATE: 08/01/2025
PERIOD OF PERFORMANCE END DATE: 07/31/2029

Group Awards By:

View Award Description

Semantic-based auditory attention decoder using Large Language Model - SUMMARY Hearing loss is one of the leading mid-life risk factors for dementia, contributing to 11% of cases globally. Dementia affects 50 million people worldwide, projected to reach 152 million by 2050, with annual costs of $1 trillion. While hearing aids and cochlear implants (CIs) are increasingly used to treat sensorineural hearing loss, these devices must do more than only amplify all the sound, including noise. They need to enhance the signal- to-noise ratio and prevent cognitive overload for understanding speech in noise (SiN). CI users show varying speech perception outcomes, especially in noisy environments, as understanding SiN involves complex sensory and cognitive processes. Even some individuals with normal hearing thresholds struggle with SiN. Difficulty in speech perception in noise can lead to social isolation, which may contribute to the development of dementia. This study proposes to start developing technology that decodes how the brain processes meaning with the listener's attention by using the existing datasets of electroencephalography (EEG), a method that measures brain signals. This technology will be able to enhance the sound of the target speaker in a conversation by using the decoded attention while filtering out unwanted speech with noise. Previous research on EEG attentional decoding focused on speech envelope, derived from the overall shape of the speech amplitude over time, neglecting key elements like meaning and context, which are crucial cues in real-world environments. To better understand and decode the brain when we listen to speech in noise, we should also involve brain areas responsible for semantic processing, which will lead to effective neuro-driven hearing amplification. Building on earlier work involving continuous language reconstruction from functional Magnetic Resonance Imaging (fMRI) recordings, our approach separates speech signals in noisy settings and decodes which one is being focused on, achieving the same result as natural language decoding but with potentially greater accuracy and effectiveness. EEG offer higher temporal resolution than fMRI, allowing the interpretation of about two words per second, while requiring feature extraction from multiple brain regions involved in semantic processing A large language model (LLM) will predict the brain's semantic processing features from transcribed conversations. A decision model will then compare these predicted features with actual brain data to identify the conversation that best matches the current semantic processing, determining which conversation is being attended to. The study has two key aims: Aim 1 is to develop algorithms that separate speech signals and cluster transcribed conversations, mimicking auditory object formation in brain. Aim 2 focuses on LLM-based attention decoding to extract semantic processing features from EEG data and clustered conversations, to identify the attended conversation. This research will provide the foundation for future neuro-steered hearing devices to assist people with hearing loss or CAPD. Dementia can impair cognition through auditory processing issues. Our proposed technology, which supports auditory neural circuits, may enhance social interactions and help prevent dementia.


Issue Date FY	Funding FY	Legal Entity Name	Legal Entity Address	Legal Entity City	Legal Entity State	Legal Entity Zip Code	Legal Entity COUNTY	Legal Entity COUNTRY	Assistance Listing	Award Code	Budget Year	Action Date	Action Type	Action Amount

Issue Date FY: 2026 ( Subtotal = $185,000 )
2026	2026	ROCHESTER INSTITUTE OF TECHNOLOGY	1 LOMB MEMORIAL DR	ROCHESTER	NY	14623	MONROE	USA	Biomedical Research and Research Training	000	2	7/14/2026	NON-COMPETING CONTINUATION	$185,000
														Subtotal = $185,000

Issue Date FY: 2025 ( Subtotal = $185,000 )
2025	2025	ROCHESTER INSTITUTE OF TECHNOLOGY	1 LOMB MEMORIAL DR	ROCHESTER	NY	14623	MONROE	USA	Biomedical Research and Research Training	000	1	7/25/2025	NEW	$185,000
														Subtotal = $185,000

Grand Total All Awards = $370,000

Top

All Categories

About

Search

Reports

Data Submission

Award Information

Semantic-based auditory attention decoder using Large Language Model

Award Number: R16GM159795

ORGANIZATION: NATIONAL INSTITUTE OF GENERAL MEDICAL SCIENCES

OPDIV: NIH

AWARD CLASS: DISCRETIONARY

AWARD ACTIVITY TYPE: SCIENTIFIC/HEALTH RESEARCH (INCLUDES SURVEYS)

PERIOD OF PERFORMANCE START DATE: 08/01/2025

PERIOD OF PERFORMANCE END DATE: 07/31/2029

Federal Websites

Department of Health & Human Services

HHS Operating Divisions

HHS Staff Divisions

Download A Document Viewer