A Parallel File System with Application-Aware Data Layout Policies for Massive Remote Sensing Image Processing in Digital Earth

Wang, Lizhe; Ma, Yan; Zomaya, Albert Y.; Ranjan, Rajiv; Chen, Dan

doi:10.1109/tpds.2014.2322362

Cited by 63 publications

(27 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In addition to the conventional continuous data access request, the data access request also has irregular data access modes such as column access and rectangular block access. The difficulty of this data access mode is that the algorithm usually needs to access a large number of pieces of data with a variety of spatial spacing and size on the file view [19].…”

Section: Access Of Image Datamentioning

confidence: 99%

A Distributed Storage Scheme for Remote Sensing Image based on Mapfile

Chen¹,

Nie²,

Jing³

2018

IJPE

View full text Add to dashboard Cite

Hyperspectral image has a large amount of data and complex structure. The distributed storage of massive remote sensing data is a hot topic today; however, the current research mostly separates the image pixels and metadata, resulting in poor system cohesion and poor data access performance. At the same time, the needs of various upper-level remote sensing algorithms are not fully considered, which makes the system less available. In view of the above problems, this paper presents a distributed image storage model based on HDFS, which stores the entire image data model in a structure to improve the system cohesion, and provides a flexible data blocking strategy for upper-level applications to meet a variety of data access needs. The comparison experiments show that the storage model has better access performance than the existing schemes.

show abstract

Section: Access Of Image Datamentioning

confidence: 99%

A Distributed Storage Scheme for Remote Sensing Image based on Mapfile

Chen¹,

Nie²,

Jing³

2018

IJPE

View full text Add to dashboard Cite

show abstract

“…This is achieved by computing a splitting scheme as described in Section II B, and determining the way of Most of RS processing applications produce images in raster format. Unfortunately, this kind of data have large size, leading to an I/O (Input/Output) bottleneck [14]. For this purpose, we develop a mapper able to write GeoTiff file on parallel file systems.…”

Section: Parallelized Pipelinementioning

confidence: 99%

A Generic Framework for the Development of Geospatial Processing Pipelines on Clusters

Cresson

Hautreux

2016

IEEE Geosci. Remote Sensing Lett.

View full text Add to dashboard Cite

The amount of remote sensing data available to applications is constantly growing due to the rise of very-high-resolution sensors and short repeat cycle satellites. Consequently, tackling computational complexity in Earth Observation information extraction is rising as a major challenge. Resorting to High Performance Computing (HPC) is becoming a common practice, since it provides environments and programming facilities able to speed-up processes. In particular, clusters are flexible, cost-effective systems able to perform data-intensive tasks ideally fulfilling any computational requirement. However, their use typically implies a significant coding effort to build proper implementations of specific processing pipelines. This paper presents a generic framework for the development of RS images processing applications targeting cluster computing. It is based on common open sources libraries, and leverages the parallelization of a wide variety of image processing pipelines in a transparent way. Performances on typical RS tasks implemented using the proposed framework demonstrate a great potential for the effective and timely processing of large amount of data.

show abstract

“…Parallel computing technologies have been recommended in the geosciences. Various techniques for the high-performance parallel processing and analysis of remote sensing data and geospatial datasets based on clusters have been proposed [27][28][29]. Geospatial applications based on parallel computing have also been designed and implemented [25,30].…”

Section: Parallel Computingmentioning

confidence: 99%

Building an Elastic Parallel OGC Web Processing Service on a Cloud-Based Cluster: A Case Study of Remote Sensing Data Processing Service

Tan

Deng

et al. 2015

Sustainability

View full text Add to dashboard Cite

Since the Open Geospatial Consortium (OGC) proposed the geospatial Web Processing Service (WPS), standard OGC Web Service (OWS)-based geospatial processing has become the major type of distributed geospatial application. However, improving the performance and sustainability of the distributed geospatial applications has become the dominant challenge for OWSs. This paper presents the construction of an elastic parallel OGC WPS service on a cloud-based cluster and the designs of a high-performance, cloud-based WPS service architecture, the scalability scheme of the cloud, and the algorithm of the elastic parallel geoprocessing. Experiments of the remote sensing data processing service demonstrate that our proposed method can provide a higher-performance WPS service OPEN ACCESSSustainability 2015, 7 14246 that uses less computing resources. Our proposed method can also help institutions reduce hardware costs, raise the rate of hardware usage, and conserve energy, which is important in building green and sustainable geospatial services or applications.

show abstract

A Parallel File System with Application-Aware Data Layout Policies for Massive Remote Sensing Image Processing in Digital Earth

Cited by 63 publications

References 10 publications

A Distributed Storage Scheme for Remote Sensing Image based on Mapfile

A Distributed Storage Scheme for Remote Sensing Image based on Mapfile

A Generic Framework for the Development of Geospatial Processing Pipelines on Clusters

Building an Elastic Parallel OGC Web Processing Service on a Cloud-Based Cluster: A Case Study of Remote Sensing Data Processing Service

Contact Info

Product

Resources

About