Building Reliable Vision-Language Assistant for Dermatology AI through Modeling Uncertainties in Multimodal LLMs - PROJECT SUMMARY Diagnosis delay is one of the key factors that lead to skin cancer death, especially for melanoma diagnosis during the COVID-19 pandemic. The long examination time and limited dermatological access have been the major roadblocks to the preventive treatment of skin cancers to lower the high mortality rate. Developing a clinical AI agent that can analyze digital skin images and provide timely, interactive text responses to patient symptoms and inquiries will significantly mitigate the nationwide dermatologist shortage, thereby improving the early diagnosis chance and teledermatology accessibility for melanoma as well as other skin diseases. Conventional dermatology AI methods mainly focus on medical image recognition to identify skin lesions and malignancies, falling short in visual-language assistance for remote healthcare services. A conversational diagnostic AI model, which is able to answer medical questions by sensing subtle visual patterns of skin disorders/cancers, is still in urgent need. The long-term goal of this research program is to develop a reliable large visual-language (VL) model that enables conversational Dermatology AI to facilitate early melanoma diagnosis and general skin care. The proposed research will generate accurate and interpretable clinical responses by finetuning Large Language Models – LLMs (e.g., the generative AI models deployed in ChatGPT) through answering questions and visual reasoning in multimodal contexts. Specifically, the project will realize three aims: 1) Build a new multimodal LLM specifically for dermatology to discern melanoma and other skin diseases and to automatically answer questions relevant to skin lesions. 2) Study uncertainties stemming from data bias and distribution shifts to enhance the reliability of LLM-powered AI diagnosis in multimodal contexts and teledermatology environments. 3) Determine the visual relevance in LLM decisions based on the rich public dermatological images with clinical text annotations. The proposed research will establish a new multimodal LLM that interweaves visual reasoning and uncertainties to advance Dermatology AI in broad VL assistance tasks, enabling automatic conversational diagnostic in teled- ermatology and providing new insights about how LLM understands skin lesions and dermatological knowledge. A pixelwise visual instruction tuning approach and a novel multi-level uncertainty quantification framework will be developed, providing technical foundations to benefit a wide range of LLM-based healthcare research. This project will be the first large visual-language research study that investigates LLM's intelligence and reliability in coping with multimodal dermatology contexts – visual skin lesions and text clinical annotations/dialogues. The success of this project will provide transformative AI techniques in assisting early melanoma diagnosis and remote skin care, leading to better teledermatology accessibility for patient treatments, reducing mortality from skin cancers through timely detection, and revolutionizing dermatological access in public healthcare systems.
