Official websites use .gov

A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS

A lock ( 🔒 ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Need Help

Building Reliable Vision-Language Assistant for Dermatology AI through Modeling Uncertainties in Multimodal LLMs

Award Number: R16GM159146
ORGANIZATION: NATIONAL INSTITUTE OF GENERAL MEDICAL SCIENCES
OPDIV: NIH
AWARD CLASS: DISCRETIONARY
AWARD ACTIVITY TYPE: SCIENTIFIC/HEALTH RESEARCH (INCLUDES SURVEYS)
PERIOD OF PERFORMANCE START DATE: 08/12/2025
PERIOD OF PERFORMANCE END DATE: 06/30/2029

Group Awards By:

View Award Description

Building Reliable Vision-Language Assistant for Dermatology AI through Modeling Uncertainties in Multimodal LLMs - PROJECT SUMMARY Diagnosis delay is one of the key factors that lead to skin cancer death, especially for melanoma diagnosis during the COVID-19 pandemic. The long examination time and limited dermatological access have been the major roadblocks to the preventive treatment of skin cancers to lower the high mortality rate. Developing a clinical AI agent that can analyze digital skin images and provide timely, interactive text responses to patient symptoms and inquiries will signiﬁcantly mitigate the nationwide dermatologist shortage, thereby improving the early diagnosis chance and teledermatology accessibility for melanoma as well as other skin diseases. Conventional dermatology AI methods mainly focus on medical image recognition to identify skin lesions and malignancies, falling short in visual-language assistance for remote healthcare services. A conversational diagnostic AI model, which is able to answer medical questions by sensing subtle visual patterns of skin disorders/cancers, is still in urgent need. The long-term goal of this research program is to develop a reliable large visual-language (VL) model that enables conversational Dermatology AI to facilitate early melanoma diagnosis and general skin care. The proposed research will generate accurate and interpretable clinical responses by ﬁnetuning Large Language Models – LLMs (e.g., the generative AI models deployed in ChatGPT) through answering questions and visual reasoning in multimodal contexts. Speciﬁcally, the project will realize three aims: 1) Build a new multimodal LLM speciﬁcally for dermatology to discern melanoma and other skin diseases and to automatically answer questions relevant to skin lesions. 2) Study uncertainties stemming from data bias and distribution shifts to enhance the reliability of LLM-powered AI diagnosis in multimodal contexts and teledermatology environments. 3) Determine the visual relevance in LLM decisions based on the rich public dermatological images with clinical text annotations. The proposed research will establish a new multimodal LLM that interweaves visual reasoning and uncertainties to advance Dermatology AI in broad VL assistance tasks, enabling automatic conversational diagnostic in teled- ermatology and providing new insights about how LLM understands skin lesions and dermatological knowledge. A pixelwise visual instruction tuning approach and a novel multi-level uncertainty quantiﬁcation framework will be developed, providing technical foundations to beneﬁt a wide range of LLM-based healthcare research. This project will be the ﬁrst large visual-language research study that investigates LLM's intelligence and reliability in coping with multimodal dermatology contexts – visual skin lesions and text clinical annotations/dialogues. The success of this project will provide transformative AI techniques in assisting early melanoma diagnosis and remote skin care, leading to better teledermatology accessibility for patient treatments, reducing mortality from skin cancers through timely detection, and revolutionizing dermatological access in public healthcare systems.


Issue Date FY	Funding FY	Legal Entity Name	Legal Entity Address	Legal Entity City	Legal Entity State	Legal Entity Zip Code	Legal Entity COUNTY	Legal Entity COUNTRY	Assistance Listing	Award Code	Budget Year	Action Date	Action Type	Action Amount

Issue Date FY: 2025 ( Subtotal = $175,470 )
2025	2025	ROCHESTER INSTITUTE OF TECHNOLOGY	1 LOMB MEMORIAL DR	ROCHESTER	NY	14623	MONROE	USA	Biomedical Research and Research Training	000	1	8/12/2025	NEW	$175,470
														Subtotal = $175,470

Grand Total All Awards = $175,470

Top

All Categories

About

Search

Reports

Data Submission

Award Information

Building Reliable Vision-Language Assistant for Dermatology AI through Modeling Uncertainties in Multimodal LLMs

Award Number: R16GM159146

ORGANIZATION: NATIONAL INSTITUTE OF GENERAL MEDICAL SCIENCES

OPDIV: NIH

AWARD CLASS: DISCRETIONARY

AWARD ACTIVITY TYPE: SCIENTIFIC/HEALTH RESEARCH (INCLUDES SURVEYS)

PERIOD OF PERFORMANCE START DATE: 08/12/2025

PERIOD OF PERFORMANCE END DATE: 06/30/2029

Federal Websites

Department of Health & Human Services

HHS Operating Divisions

HHS Staff Divisions

Download A Document Viewer