OME-NGFF: scalable format strategies for interoperable bioimaging data

Abstract

Biological imaging is one of the most innovative fields in the modern biological sciences. New imaging modalities, probes, and analysis tools appear every few months and often prove decisive for enabling new directions in scientific discovery. One feature of this dynamic field is the need to capture new types of data and data structures. While there is a strong drive to make scientific data Findable, Accessible, Interoperable and Reproducible (FAIR, 1), the rapid rate of innovation in imaging impedes the unification and adoption of standardized data formats. Despite this, the opportunities for sharing and integrating bioimaging data and, in particular, linking these data to other "omics" datasets have never been greater; therefore, to every extent possible, increasing "FAIRness" of bioimaging data is critical for maximizing scientific value, as well as for promoting openness and integrity. In the absence of a common, FAIR format, two approaches have emerged to provide access to bioimaging data: translation and conversion. On-the-fly translation produces a transient representation of bioimage metadata and binary data but must be repeated on each use. In contrast, conversion produces a permanent copy of the data, ideally in an open format that makes the data more accessible and improves performance and parallelization in reads and writes. Both approaches have been implemented successfully in the bioimaging community but both have limitations. At cloud-scale, those shortcomings limit scientific analysis and the sharing of results. We introduce here next-generation file formats (NGFF) as a solution to these challenges.

Towards community-driven metadata standards for light microscopy: tiered specifications extending the OME model

“…To this aim, in addition to our work in the context of QUAREP-LiMi, the further development of the Microscopy Metadata Specifications is being coordinated with other parallel initiatives including: 1) the OME community development of general criteria and procedures to capture and store metadata in OME-NGFF (Text Box 1). The OME NGFF effort (35,36) is implementing storage approaches to hold the binary pixel data and the metadata described herein in standardized, shareable, long-lived, efficient, and performant containers (e.g. files).…”

“…2) Shared rules for how the (ideally) automated capture, representation and storage of Microscopy Metadata should be implemented in practice (yellow bubble). Last but not least, 3) Next-Generation File Formats (NGFF) where the ever-increasing scale and complexity of image data and metadata would be contained for exchange (35,36); blue bubble). All these provisions must be compatible with the FAIR principles of data stewardship (22), maximize consistency, flexibility, extensibility, and adapt to ever-growing experimental and technical complexity.…”

“…Specifically, the OME Data Model initially published in 2005 (4, 5) and its related software implementations (4, 72) were developed to provide an informatics framework for the accurate capture, storage and management of rich metadata representations of all the information required for interpreting, comparing, reproducing, sharing and publishing biological microscopy data (Figure 2, Microscopy Metadata) independent of a proprietary file format. The OME Data Model is first and foremost implemented in the Bio-Formats library (4) which allows the interpretation of the more than 150 different proprietary file formats and their translation into the OME-TIFF and OME-NGFF (35,36) open file formats for biological imaging. This facilitates the transfer of image data between different commercial vendors and open-source software tools thus reducing the barriers between different analytical and data management tools.…”

“…Specifically, it puts forth scalable specifications for light Microscopy Metadata developed jointly by the 4D Nucleome Initiative (4DN) (1, 2) Imaging Standards Working Group (IWG) and by the BioImaging North America (BINA) Quality Control and Data Management Working Group (QC-DM-WG) (3,7,8). In order to foster widespread adoption of these 4DN-BINA-OME (NBO namespace) (26) specifications framework (Figure 1A, magenta bubble), parallel work is being conducted to develop userfriendly and when possible automated metadata collection tools (13,(27)(28)(29)(30); shared metadata storage best practices (Figure 1A, yellow bubble) (31)(32)(33)(34); as well as sustainable specifications to switch from proprietary formats for image data into common, cloud-ready OME Next-Generation File Formats (NGFF, Figure 1A, blue bubble) (35,36). Importantly, all of these activities are carried out in the context of the newly launched QUAlity Assessment and REProducibility for Instrument and Images in Light Microscopy (QUAREP-LiMi) initiative (quarep.org; (9,37) and involve several key members of the community, including microscope users, custodians, and manufacturers, imaging scientists, national and global bioimaging organizations, bio-image informaticians, standards organizations, and scientific publishers.…”

“…The establishment of community-driven Microscopy Image Data Standards implies parallel development on three interrelated fronts: 1) Community-driven specifications for what 'data provenance' information and quality control metrics are essential for rigor, reproducibility, and reuse and should therefore be captured in Microscopy Metadata (magenta bubble). 2) Shared rules for how the (ideally) automated capture, representation and storage of Microscopy Metadata should be implemented in practice (yellow bubble).Last but not least, 3) Next-Generation File Formats (NGFF) where the ever-increasing scale and complexity of image data and metadata would be contained for exchange(35,36); blue bubble). All these provisions must be compatible with the FAIR principles of data stewardship(22), maximize consistency, flexibility, extensibility, and adapt to ever-growing experimental and technical complexity.…”