Big Data Integration in Remote Sensing across a Distributed Metadata-Based Spatial Infrastructure

Abstract

Since Landsat-1 first started to deliver volumes of pixels in 1972, the volumes of archived data in remote sensing data centers have increased continuously. Due to various satellite orbit parameters and the specifications of different sensors, the storage formats, projections, spatial resolutions, and revisit periods of these archived data are vastly different. In addition, the remote sensing data received continuously by each data center arrives at a faster code rate; it is best to ingest and archive the newly received data to ensure users have access to the latest data retrieval and distribution services. Hence, an excellent data integration, organization, and management program is urgently needed. However, the multi-source, massive, heterogeneous, and distributed storage features of remote sensing data have not only caused difficulties for integration across distributed data center spatial infrastructures, but have also resulted in the current modes of data organization and management being unable meet the rapid retrieval and access requirements of users. Hence, this paper proposes an object-oriented data technology (OODT) and SolrCloud-based remote sensing data integration and management framework across a distributed data center spatial infrastructure. In this framework, all of the remote sensing metadata in the distributed sub-centers are transformed into the International Standardization Organization (ISO) 19115-based unified format, and then ingested and transferred to the main center by OODT components, continuously or at regular intervals. In the main data center, in order to improve the efficiency of massive data retrieval, we proposed a logical segmentation indexing (LSI) model-based data organization approach, and took SolrCloud to realize the distributed index and retrieval of massive metadata. Finally, a series of distributed data integration, retrieval, and comparative experiments showed that our proposed distributed data integration and management program is effective and promises superior results. Specifically, the LSI model-based data organization and the SolrCloud-based distributed indexing schema was able to effectively improve the efficiency of massive data retrieval.

Stewardship and analysis of big Earth observation data

“…However, in order to obtain such valuable information, massive numbers of remote sensing images have to processed, a process which is similar to trying to extract gold from sand. (5) HeterogeneousDue to the great variety in satellite orbit parameters and the specifications of sensors, the storage formats, projections, spatial resolutions, and revisit periods of archived data also vary enormously, and these differences have resulted in great difficulties for data stewardship and management (Fan, Yan, Ma, & Wang, 2017).…”

Enabling the Big Earth Observation Data via Cloud Computing and DGGS: Opportunities and Challenges

“…In terms of efficiency of EO data processing, Figure 5 shows that cloud computing provides some services for big Earth observation data (BEOD), including spatial data infrastructure (SDI), EO data resource, algorithm or model library, processing and computation, systems and applications [11,45,46]. For example, the parallel mosaic and interpretation algorithms based on cloud computing have advantages in efficiency [47,48]. The most straightforward pattern is to provide spatial data infrastructures, such as AWS, Google Cloud, and Aliyun, which have a large number of clusters, machines, and servers that can provide infrastructure level services for users on-demand.…”

Convolutional Neural Network with Expert Knowledge for Hyperspectral Remote Sensing Imagery Classification

“…Instead of the full connected layer in the traditional CNN structure, convolutional layer with a 1×1 convolution kernel [18] is employed to reduce the number of training parameters. L 2 regularization and Dropout strategies [19] are utilized during training process in order to improve the generalization of the CNN network. L 2 regularization is able to minimize the cost function in the training stage by making the sum of the squares of the parameter small.…”