Real-time Conversion and Display of Visual Diagrams in Accessible Forms for BVIs - PROJECT SUMMARY Visual diagrams are playing an increasingly dominant role in providing information in education, employment and daily living. Unfortunately, individuals who are blind or visually impaired (BVIs) experience enormous obstacles in independently accessing this information, impacting their success in all aspects of their lives. The proposed project intends to addresses this critical need through the development of a low-cost system for the real-time automatic conversion of visual diagrams into accessible dual vibrotactile and enhanced visual forms rendered on aligned, refreshable tactile and visual displays, allowing BVIs to use touch, their residual vision or a combination of both to access graphical information. To ease the significant time and cognitive load required by users to interpret accessible diagrams, automated haptic guidance will be available to help BVIs explore a diagram through shared control with the user. Basic alt-text descriptions will also be available, both in Braille and in spoken form, to also support the effective exploration of the diagram. A participatory design approach will be used for the system’s development, involving BVIs and other stakeholders in: focus groups, low technology prototyping and user testing both in the laboratory and in the field. The display system will consist of a low-cost refreshable haptic display, previously developed in one of the PI’s laboratories, combining multi-fingered tactile feedback with shared control of movement between the user and haptic guidance fixtures (with user adjustable weightings) to which an enhanced vision display will be added. Machine learning will be used to convert visual diagrams to an accessible form useful for both lower level and higher-level thinking. To obtain a more effective model for conversion, the key component to our approach is to take advantage of the underlying structure of known diagram types, with separate models for graphs and charts (simple and highly structured), illustrations and photos (complex and least structured), and maps and engineering diagrams (focused on the connectivity of elements and using a symbol set). The R61 Phase of the project will focus on providing a beta version of the complete system for simple, high impact graphics (graphs and charts) through multiple design iterations with BVI user testing in the laboratory. The R33 Phase of the project will deploy the beta version in the field with BVI users to further assess and improve the design for real world use. This phase will also work on the development of conversion methods for illustrations and photos, and indoor/outdoor maps and engineer diagrams, again through multiple design iterations with BVI user testing the laboratory. Dissemination will also occur during this stage.
