The impact of the data archiving file format on the sharing of scientific data for use in popular computational environments

The Impact of the Data Archiving File Format on Scientific Computing and Performance of Image Processing Algorithms in MATLAB Using Large HDF5 and XML Multimodal and Hyperspectral Data Sets

“…In most cases, the resulting XML file is between 2.5 and 3 three times as large as the similar HDF5 file. This finding is consistent with other published results (Bennett & Robertson, 2010 After saving the raw image data to HDF5 and XML files, each file was loaded and processed according to the steps shown previously in Figure 2. These steps include loading the file, adjusting the image, calculating image statistics, removing noise, detecting edges, and detecting features.…”

“…MATLAB has many advantages over conventional languages (such as FORTRAN, and C++) for scientific data analysis, such as ease of use, platform independence, device-independent plotting, graphical user interface, and the MATLAB compiler (Chapman, 2008). Previous results have shown HDF5 format provided faster load and process times than XML formats, and loads large amounts of data without running into memory issues (Bennett & Robertson, 2010). This research supports these findings.…”

“…Such analysis includes analyzing the relative performance of the file format, including scalability, with respect to various computational tools, computer architectures, and operating systems (Bennett & Robertson, 2010). In this chapter we provide insights into the challenges researchers face with a growing set of data, along with expectations for performance guidelines on workstations for processing large HDF5 and XML hyperspectral image data.…”

MATLAB - A Ubiquitous Tool for the Practical Engineer

“…In most cases, the resulting XML file is between 2.5 and 3 three times as large as the similar HDF5 file. This finding is consistent with other published results (Bennett & Robertson, 2010 Table 9. HDF5 and XML Raw Image File Size Statistics.…”

“…To assist researchers in exchanging the data needed to develop, test, and optimize the techniques, selecting the best file format for computing environments (such as MATLAB) requires additional analysis. Such analysis includes analyzing the relative performance of the file format, including scalability, with respect to various computational tools, computer architectures, and operating systems (Bennett & Robertson, 2010). In this chapter we provide insights into the challenges researchers face with a growing set of data, along with expectations for performance guidelines on workstations for processing large HDF5 and XML hyperspectral image data.…”