A Scalable Computing Resources System for Remote Sensing Big Data Processing Using GeoPySpark Based on Spark on K8s

Guo, Jingkun; Huang, Chunlin; Hou, Jinliang

doi:10.3390/rs14030521

Cited by 14 publications

(6 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides, Spark requires adjusting the programming code with the environment. The papers [19,20] use Spark-on-Kubernetes deployed on a cloud to overcome the mentioned issues. In [19], researchers suggest a task scheduling mechanism to change the worker pods dynamically.…”

Section: Motivationmentioning

confidence: 99%

“…In [19], researchers suggest a task scheduling mechanism to change the worker pods dynamically. GeoPySpark, based on Spark, provides a scalable remote sensing data processing system, where HDFS is deployed on nodes of Kubernetes to diminish data transfer between workers [20]. The limitation of the latest three works is the absence of linkage with DC, not interoperable from data repositories, as they either use HDFS or external storage.…”

Section: Motivationmentioning

confidence: 99%

See 1 more Smart Citation

Scalable Data Processing Platform for Earth Observation Data Repositories

Astsatryan

Lalayan

Giuliani

2023

SCPE

View full text Add to dashboard Cite

Earth observation (EO) satellite data is essential to environmental monitoring. At a national and regional level, the open data cubes harness the power of satellite data by providing application programming interfaces and services to the end-users. The volume and the complexity of satellite observations are increasing, demanding novel approaches for data storing, managing, and processing. High-performance computing (HPC) and cloud platforms may improve Big EO data processing performance. However, it is necessary to consider several vital aspects for efficient and flexible EO data processing, such as the interoperability from cloud-HPC and EO data repositories, automatic provisioning and scaling of cloud-HPC resources, cost-effectiveness, support of new EO data formats and open-source packages, or linkage with data cube platforms. The article proposes a scalable EO data processing platform interoperable from cloud-HPC and EO data repositories. The platform enables linking any data repository supporting web coverage service or SpatioTemporal Asset Catalog Application Programming Interfaces (STAC-API), and any cloud or HPC resource supporting scheduling system API for providing access to the cluster backends.

show abstract

Section: Motivationmentioning

confidence: 99%

Section: Motivationmentioning

confidence: 99%

Scalable Data Processing Platform for Earth Observation Data Repositories

Astsatryan

Lalayan

Giuliani

2023

SCPE

View full text Add to dashboard Cite

show abstract

“…When the main consultant wishes to invite other consultant for the purpose of prescription then the consultant should generate public and private keys. The private key of the main consultant is a random element x of Z r , and public key is g x [28]. T invite the other consultant, the main consultant hashes the invitation the message to some number h of G 1 , the gives back h x .…”

Section: Privacy Preservationmentioning

confidence: 99%

An Intelligent Cardiovascular Diseases Prediction System Focused on Privacy

Kolhar¹,

Misfer²

2023

Intelligent Automation &Amp; Soft Computing

View full text Add to dashboard Cite

Machine learning (ML) and cloud computing have now evolved to the point where they are able to be used effectively. Further improvement, however, is required when both of these technologies are combined to reap maximum benefits. A way of improving the system is by enabling healthcare workers to select appropriate machine learning algorithms for prediction and, secondly, by preserving the privacy of patient data so that it cannot be misused. The purpose of this paper is to combine these promising technologies to maintain the privacy of patient data during the disease prediction process. Treatment of heart failure may be improved and expedited with this framework. We used the following machine learning algorithms to make predictions: Logistic Regression (LR), Naive Bayes (NB), K-Nearest Neighbors (KNN), Decision Tree (DT) and Support Vector Machines (SVM). These techniques, combined with cloud computing services, improved the process of deciding whether to treat a patient with cardiac disease. Using our classifiers, we classified cardiac patients according to their features, which are grouped into single features, combinations of selected features, and all features. In experiments using all clinical features, machine learning classifiers SVM, DT, and KNN outperformed the rest, whereas in experiments using minimal clinical features, SVM and KNN were the most accurate. Internet of Things (IoT) devices allow family physicians to share diagnostic reports on the cloud in a secure manner. Ring signatures are particularly useful for verifying the integrity of data exchange. Our system keeps the physician's identity confidential from all authorized users, who can still access medical reports publicly. Our proposed mechanism has been shown to be both effective and efficient when it comes to obtaining patient reports from cloud storage.

show abstract

“…A Convolutional Neural Network (CNN)based deployment solution on resource-limited FPGA for spaceborne applications using RSBD is implemented [25]. A scalable computing resource model is developed to achieve fast processing of RSBD using a parallel distributed architecture [26]. A Spark-based adaptive real-time MapReduce data processing method that improves performance and stability is presented [27].…”

Section: Algorithms and Framework For Hpc-based Rsbd Processingmentioning

confidence: 99%

Exploring the Advancements in High-Performance Computing Paradigm for Remote Sensing Big Data Analytics

SK,

Aji S

2023

Cloud Computing and Data Science

View full text Add to dashboard Cite

The incredible growth in Remote Sensing (RS) data volume, with high spectral-spatial-temporal resolutions, has been utilized in various application domains. With the rapid advancements in modern sensor technologies, including the 3D acquisition sensors, RS data with a large variety, velocity, veracity, varied value and incredible volume are generated, leading to the Remote Sensing Big Data (RSBD). With the high availability of RSBD, we require High-Performance Computing (HPC) environments for storing and processing these High-Dimensional (HD), complex, heterogeneous and distributed data. Also, introducing Deep Learning (DL) techniques in the RS domain demands more computing power, higher memory and networking bandwidth throughput capabilities, and optimized software and libraries to deliver the required performance. Motivated by this, we explore HPC computing environments for handling RSBD across multiple application domains in this paper. With particular emphasis on architectures such as cloud-based HPC, clusters, heterogeneous networks of computers, and specialized hardware architectures like Field Programmable Gate Arrays (FPGAs) and Graphics Processing Units (GPUs), we investigate how HPC technologies are being used to process RSBD efficiently while including integrated intelligence. This critical analysis results in a multi-layered cloud-based framework for efficient RSBD processing tasks. Also, we identified several data challenges to be handled while designing HPC frameworks. The findings from the study can help researchers better understand the HPC design concepts for developing RSBD frameworks.

show abstract

A Scalable Computing Resources System for Remote Sensing Big Data Processing Using GeoPySpark Based on Spark on K8s

Cited by 14 publications

References 38 publications

Scalable Data Processing Platform for Earth Observation Data Repositories

Scalable Data Processing Platform for Earth Observation Data Repositories

An Intelligent Cardiovascular Diseases Prediction System Focused on Privacy

Exploring the Advancements in High-Performance Computing Paradigm for Remote Sensing Big Data Analytics

Contact Info

Product

Resources

About