Modernizing the Hierarchical Data Modeling Framework to Advance FAIR Data for Science - PROJECT SUMMARY The goal of this project is to enhance the sustainability and impact of the Hierarchical Data Modeling Framework (HDMF), an open-source software tool for standardizing scientific data that forms the foundation of the Neurodata Without Borders (NWB) data standard and software ecosystem for neurophysiology. Supported by the NIH BRAIN Initiative, NWB has become the leading open data standard in neurophysiology, with over 310 public datasets from more than 150 labs already available on the DANDI data archive, and more than 35 community tools supporting NWB. As the HDMF and NWB user base continues to grow, maintaining its code base has become increasingly challenging. Key HDMF features that did not exist in other tools when HDMF was initially developed are now better addressed by newer, widely used technologies. As NWB becomes increasingly adopted as the primary format for sharing neurophysiology data, and as interest grows in using HDMF for other scientific data applications, we must ensure the software remains maintainable, interfaces well with other popular technologies, and is user-friendly. The project has two main objectives: Aim 1 focuses on enhancing interoperability of HDMF by implementing support for LinkML as an additional schema language. This will simplify harmonization and exchange of HDMF-based standards and data models with other data standards and formats, enable use of ontologies and controlled terminologies in HDMF schema, support the design of more expressive and rigorous data models, and improve HDMF’s sustainability and maintainability. Aim 2 focuses on improving the usability and maintainability of the HDMF API through integration with the Pydantic library. We will update the HDMF data interfaces with Pydantic models and replace the custom type validation and documentation system in HDMF with Python type hints, Pydantic validation, and autodoc-pydantic documentation generation. These changes will also make HDMF more accessible to the broader open-source community, enabling developers to better understand the codebase and contribute new features and bug fixes more effectively. We will also create test suites to ensure the stability and compatibility of HDMF for downstream applications. These improvements will address current challenges, including the high maintenance cost of custom code in HDMF, the need to interface with modern data modeling technologies, and the demand for better usability and stability. By integrating HDMF with modern, widely used, well-tested technologies such as LinkML and Pydantic, we will ensure the sustainability of HDMF for the next 5-10 years. This modernization effort will strengthen HDMF's position as a valuable tool for standardizing scientific data while enhancing the infrastructure that supports the NWB neurophysiology data standard.
