MIX-RS: A Multi-Indexing System Based on HDFS for Remote Sensing Data Storage

Abstract

A large volume of Remote Sensing (RS) data has been generated with the deployment of satellite technologies. The data facilitate research in ecological monitoring, land management and desertification, etc.The characteristics of RS data (e.g., enormous volume, large single-file size, and demanding requirement of fault tolerance) make the Hadoop Distributed File System (HDFS) an ideal choice for RS data storage as it is efficient, scalable, and equipped with a data replication mechanism for failure resilience. To use RS data, one of the most important techniques is geospatial indexing. However, the large data volume makes it time-consuming to efficiently construct and leverage. Considering that most modern geospatial data centres are equipped with HDFS-based big data processing infrastructures, deploying multiple geospatial indices becomes natural to optimise the efficacy.Moreover, because of the reliability introduced by high-quality hardware and the infrequently modified property of the RS data, the use of multi-indexing will not cause large overhead. Therefore, we design a framework called Multi-IndeXing-RS (MIX-RS) that unifies the multi-indexing mechanism on top of the HDFS with data replication enabled for both fault tolerance and geospatial indexing efficiency. Given the fault tolerance provided by the HDFS, RS data are structurally stored inside for faster geospatial indexing. Additionally, multi-indexing enhances efficiency.The proposed technique naturally sits on top of the HDFS to form a holistic framework without incurring severe overhead or sophisticated system implementation efforts. The MIX-RS framework is implemented and evaluated using real remote sensing data provided by the Chinese Academy of Sciences, demonstrating excellent geospatial indexing performance.

Data Lake Conceptualized Web Platform for Food Research Data Collection

“…Studies on the storage and processing of large datasets have been conducted in various fields. Examples include research on storing large-scale meteorological data [9], storing remotely sensed data that are produced in large quantities [10], using Hadoop for storing resource description framework models for linked data and knowledge graphs [11,12], and research on using Hadoop for processing medical data to predict chronic kidney diseases in the context of large-scale bioscience data [13]. Other studies have explored the advantages of Hadoop in storing and searching proteomic datasets [14], as well as storing largescale genomic data in FASTA/Q files [15].…”

A Fast Large-Scale Path Planning Method on Lunar DEM Using Distributed Tile Pyramid Strategy

“…Compared with a single computer, distributed storage (e.g., Hadoop distributed file system (HDFS), HBase) and computational technologies (e.g., MapReduce, Spark) use the storage and computational resources of clusters and show tremendous advantages when data increases dramatically. Therefore, they are extensively used in the storage [19][20][21], calculation [22,23], segmentation [24], and path planning of massive remote sensing data. Wang et al used the MapReduce-based distributed parallel Dijkstra algorithm to solve the shortest path problem.…”

How does solid‐state drives cluster perform for distributed file systems: An empirical study

“…Thus, a hybrid DFS, which exploits a bunch of SSDs to work as a small cluster (i.e., the SSD cluster in our particular sense) to facilitate the storage system as a whole is more practical in reality 7,8 . For example, the SSD cluster is in practice often used to optimise the storage of small data 9,10 , metadata 11,12 or functioning as caches for hot data 13 under common scenarios such as Internet of Things (IoT) 14,15 .…”