VS-EDGE: Visual-Semantic Explanations for Diagnostic Guidance - PROJECT SUMMARY Technological innovations such as computer-aided diagnosis (CAD) systems can be of value in analyzing the sixty billion radiological images that are generated annually in America’s healthcare sector. However, there is a large gap between the number of CAD systems reported in the scientific literature and the number routinely used in clinical practice. This gap can be partially explained by the fact that most machine learning-based CAD systems function in a “black box” manner, leading to a lack of trust by clinicians in the use of technology to aid the diagnostic process in clinical settings. Recognizing an object involves rapid visual processing and activation of semantic knowledge about the object, but how visual processing activates and interacts with semantic representations for both physician and CAD systems remains unclear. There is, therefore, a critical need to determine how visual-semantic processing identifies important, clinically significant visual patterns. The long-term goal is to accelerate the development of technological innovations for improved decision making in clinical settings. The overall objective for the proposed R15 is to develop and validate a novel model of visual-semantic processing. The central hypothesis is that a model based on an integrated semantic and deep-learning neural network (SDNN) can create explainable and accurate CAD mechanisms and establish a common understanding of what is visually important to radiologists. The rationale is that successful development of a validated explainable CAD model will provide new opportunities for its continued development and future clinical use as an aid to physicians making diagnoses. The following two specific aims are proposed: 1) Develop a Semantic Deep-learning Neural Network (SDNN) that generates visual patterns and image exemplars; and 2) Determine which visual patterns and image exemplars are diagnostically most informative. In the first aim, a deep learning model will be designed and developed that, when augmented with visual semantic knowledge, gives rise to representational features that closely match the visual semantic concepts represented in human higher-level vision. For the second aim, a panel of experts will be engaged to identify a set of visual patterns and image exemplars that can be used as potential CAD explanations when interpreting lung nodules by computer. The proposed research is innovative because it focuses on integrating visual perception and semantic knowledge and discovering emergent patterns that more closely match the visual-semantic features represented in human higher-level visual cortex. This proposed research is significant because the results are expected to provide strong scientific justification for continued development of future CAD systems augmented with explanations as a gateway between CAD technology and clinical practice, as well as continued expansion of use of these technologies. Finally, as an R15, this award will be used to help strengthen the research environment at DePaul University, as well as expose undergraduate and graduate students to biomedical research, thereby broadening their range of career choices.
