Official websites use .gov

A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS

A lock ( 🔒 ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Need Help

From Human-Powered to Automated Video Description for Blind and Low Vision Users

Award Number: R01EY034562
ORGANIZATION: NATIONAL EYE INSTITUTE
OPDIV: NIH
AWARD CLASS: DISCRETIONARY
AWARD ACTIVITY TYPE: SCIENTIFIC/HEALTH RESEARCH (INCLUDES SURVEYS)
PERIOD OF PERFORMANCE START DATE: 07/01/2023
PERIOD OF PERFORMANCE END DATE: 06/30/2028

Group Awards By:

View Award Description

From Human-Powered to Automated Video Description for Blind and Low Vision Users - Project Summary Approximately 12 million people in the United States have been diagnosed with a visual impairment. These individuals face unique challenges in our modern environment, where much critical information related to education, employment, entertainment, and community is presented in the form of digital videos. Inaccessible information can result in social exclusion or become life threatening if individuals require access to it in order to make decisions related to their health and safety. For example, in a personal or global health crisis, individuals may need to access the mass amounts of information conveyed via videos or dynamic infographics in order to make informed decisions. To address this need, the online platform YouDescribe allows blind and low vision (BLV) users to request amateur volunteers to create video descriptions, also referred to as audio descriptions (AD), of YouTube videos. However, the platform has been unable to keep up with the overwhelming demand, and 92.5% of videos on the YouDescribe user wish list remain undescribed. The overall objective of this proposal is to build an AI-driven system, suitable for use on a wide-scale, to automatically generate descriptions of online videos, as well as answer questions asked by BLV users about the content of videos. The rationale for this project is that AI-based tools are necessary to facilitate timely access to the deluge of new videos appearing on the Internet every day. The proposed work encompasses three specific aims: 1) develop an AI-based tool in collaboration with sighted describers that more efficiently produces video descriptions and increases the availability of accessible videos. The goal is to create an AI-driven NarrationBot that will decrease the time required for novice volunteers to produce video descriptions by 80%; 2) develop an AI-based tool in collaboration with BLV individuals that offers user-driven access to visual information in online videos. The goal is to develop an AI-driven QABot that allows users to pause a video, ask questions about content, and receive immediate answers (e.g., “What breed is the dog?”, “German shepherd”) that are accurate 80% of the time; and 3) develop and publicly release large-scale datasets to improve machine learning for video accessibility. These novel datasets will be used to increase the quality and accuracy of NarrationBot and QABot until AI-generated descriptions and answers need minimal intervention from human volunteers and can serve BLV users directly. The proposed research is innovative because it focuses on videos, whereas existing AI-driven efforts to address this problem have focused primarily on static photos or images. It is also one of only a few efforts to directly partner with BLV individuals to develop AI-driven systems that produce visual descriptions or answer visual questions. The proposed research is significant because it will result in open-source, AI-driven tools that will give BLV individuals unprecedented control over their ability to independently navigate the information-rich world of online videos, thus improving their health and wellbeing.


Issue Date FY	Funding FY	Legal Entity Name	Legal Entity Address	Legal Entity City	Legal Entity State	Legal Entity Zip Code	Legal Entity COUNTY	Legal Entity COUNTRY	Assistance Listing	Award Code	Budget Year	Action Date	Action Type	Action Amount

Issue Date FY: 2025 ( Subtotal = $536,828 )
2025	2025	ARIZONA STATE UNIVERSITY	660 S MILL AVENUE STE 204	TEMPE	AZ	85281	MARICOPA	USA	Vision Research	001	3	9/19/2025	NON-COMPETING CONTINUATION	$536,828
2025	2024	ARIZONA STATE UNIVERSITY	660 S MILL AVENUE STE 204	TEMPE	AZ	85281	MARICOPA	USA	Vision Research	000	2	12/5/2024	NON-COMPETING CONTINUATION	$0
														Subtotal = $536,828

Issue Date FY: 2024 ( Subtotal = $625,357 )
2024	2024	ARIZONA STATE UNIVERSITY	660 S MILL AVENUE STE 204	TEMPE	AZ	85281	MARICOPA	USA	Vision Research	001	2	8/29/2024	NON-COMPETING CONTINUATION	$0
2024	2024	ARIZONA STATE UNIVERSITY	660 S MILL AVENUE STE 204	TEMPE	AZ	85281	MARICOPA	USA	Vision Research	000	2	7/5/2024	NON-COMPETING CONTINUATION	$625,357
														Subtotal = $625,357

Issue Date FY: 2023 ( Subtotal = $649,872 )
2023	2023	ARIZONA STATE UNIVERSITY	660 S MILL AVENUE STE 204	TEMPE	AZ	85281	MARICOPA	USA	Vision Research	002	1	8/28/2023	NEW	$0
2023	2023	ARIZONA STATE UNIVERSITY	660 S MILL AVENUE STE 204	TEMPE	AZ	85281	MARICOPA	USA	Vision Research	000	1	6/15/2023	NEW	$582,178
2023	2023	ARIZONA STATE UNIVERSITY	660 S MILL AVENUE STE 204	TEMPE	AZ	85281	MARICOPA	USA	Vision Research	001	1	8/21/2023	NEW	$67,694
														Subtotal = $649,872

Grand Total All Awards = $1,812,057

Top

All Categories

About

Search

Reports

Data Submission

Award Information

From Human-Powered to Automated Video Description for Blind and Low Vision Users

Award Number: R01EY034562

ORGANIZATION: NATIONAL EYE INSTITUTE

OPDIV: NIH

AWARD CLASS: DISCRETIONARY

AWARD ACTIVITY TYPE: SCIENTIFIC/HEALTH RESEARCH (INCLUDES SURVEYS)

PERIOD OF PERFORMANCE START DATE: 07/01/2023

PERIOD OF PERFORMANCE END DATE: 06/30/2028

Federal Websites

Department of Health & Human Services

HHS Operating Divisions

HHS Staff Divisions

Download A Document Viewer