Toward Artificial Intelligence-enabled Automatic Echocardiography Reporting: A Study of Visual-Linguistic Foundation Model Frameworks - PROJECT SUMMARY/ABSTRACT As the primary non-invasive imaging tool in cardiology, echocardiography delays and misdiagnoses significantly affect outcomes for the 5.5% of the U.S. population with cardiac issues. Automation of echocardiography interpretation is proposed but remains limited to models with narrow image analysis functions. Research into Visual-Linguistic Models (VLMs) and Large Language Models (LLMs) to build an Artificial Intelligence-enabled Echocardiography Interpretation System (AEIS) presents a solution to this problem and paves the way to next- generation Generalist Medical AI (GMAI) systems. In this proposal, Dr. Chieh-Ju Chao is seeking a K08 Mentored Career Development Award to advance his training to develop and evaluate an AEIS. The mentoring team is led by Drs. Bradley Erickson (primary mentor) and Fei-Fei Li (co-primary mentor), both are worldclass experts with complementary AI ability in medical imaging and computer vision. They are funded by agencies including NIH, NSF, and DoD, with a proven record of accomplishment of successfully mentoring junior investigators. The Research Training Plan will equip Dr. Chao with advanced skills through mentorship from a cross-site, multidisciplinary team, with Dr. Bradley Erickson serving as the central figure to ensure training quality. In Aim 1. Dr. Chao will train with Drs. Fei-Fei Li and Ehan Adeli to pretrain VLMs using PubMed Open-Access figures and the CheXpert Plus datasets, followed by fine-tuning with Mayo's echocardiography data to enhance caption accuracy. Aim 2. involves training with Dr. Curtis Langlotz to optimize LLMs for factual correctness and expert preferences, and Dr. Erickson for quantitative hallucination detection to build an LLM system for echocardiography report summarization. In Aim 3, Dr. Chao will train with Dr. Imon Banerjee on multi-modal framework organization and evaluation to assemble the AEIS, and with Dr. Jae Oh on designing clinical reader studies, and Dr. Garvan Kane on AI-based workflow optimization to study the effectiveness and human-AI interaction of the AEIS. Dr. Chao has a solid foundation in echocardiography and AI, 13 publications in the medical AI field, including 6 as a first or co-first author since he entered in 2021. The Career Development Plan will enhance Dr. Chao's ability on advanced AI research skills for developing and evaluating an AEIS, with career skills gained through practical experience and coursework. The environment at Mayo Clinic, enhanced by resources from Stanford's Institute of Human-centered Artificial Intelligence through our collaborative program, is outstanding for innovative research. Excellent infrastructure (datasets, cloud and local computing resources and other equipment) is available. In summary, the robust mentoring team and training plan are expected to thoroughly prepare Dr. Chao for an independent career centered on developing and evaluating future GMAI systems in cardiovascular medicine, to improve efficiency and quality of echocardiography interpretation, as well as patient outcomes and longevity.
