The Impact of the Data Archiving File Format on Scientific Computing and Performance of Image Processing Algorithms in MATLAB Using Large HDF5 and XML Multimodal and Hyperspectral Data Sets

Abstract

146and plasma physics, astronomy, medicine and bio-imaging (Shasharina et al., 2007;Dougherty et al., 2009). In this chapter, we will use hyperspectral images stored in XML and HDF5 format to compare the relative performance of the file format using computationally intensive signal and image processing algorithms running in MATLAB on Windows ® 64-bit and Linux 64-bit workstations. Hyperspectral imaging refers to the multidimensional character of the spectral data set, where the acquisition of images takes place over many contiguous spectral bands throughout the visible and infrared (IR) regions (Goetz et al., 1985). Sensor fusion and advanced image processing techniques are now possible using the information from these different bands that allow applications in aerospace, defense, medicine, and other fields of study.To assist researchers in exchanging the data needed to develop, test, and optimize the techniques, selecting the best file format for computing environments (such as MATLAB) requires additional analysis. Such analysis includes analyzing the relative performance of the file format, including scalability, with respect to various computational tools, computer architectures, and operating systems (Bennett & Robertson, 2010). In this chapter we provide insights into the challenges researchers face with a growing set of data, along with expectations for performance guidelines on workstations for processing large HDF5 and XML hyperspectral image data. Additionally, in this chapter, we provide specific results comparing data load, process, and memory usage for the differing data formats, along with detailed discussions and implications for researchers.