Exploring cloud-based Web Processing Service: A case study on the implementation of CMAQ as a Service

Zhang, Chen; Di, Liping; Sun, Ziheng; Lin, Li; Yu, Genong; Gaigalas, Juozas

doi:10.1016/j.envsoft.2018.11.019

Cited by 32 publications

(11 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Two theory-based air quality models have been widely accepted around the industry: the community multiscale air quality (CMAQ) [10] modeling system and the comprehensive air quality model with extensions (CAMx) [11]. Both were developed based on the assumption of "one atmosphere."…”

Section: Eory-based Methodsmentioning

confidence: 99%

Longer Time Span Air Pollution Prediction: The Attention and Autoencoder Hybrid Learning Model

Zhang

Jin

Zou

et al. 2021

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Air pollution has become a critical issue in human’s life. Predicting the changing trends of air pollutants would be of great help for public health and natural environments. Current methods focus on the prediction accuracy and retain the forecasting time span within 12 hours. Shorter time span decreases the practicability of these perditions, even with higher accuracy. This study proposes an attention and autoencoder (A&A) hybrid learning approach to obtain a longer period of air pollution changing trends while holding the same high accuracy. Since pollutant concentration forecast highly relates to time changing, quite different from normal prediction problems like autotranslation, we integrate “time decay factor” into the traditional attention mechanism. The time decay factor can alleviate the impact of the value observed from a longer time before while increasing the impact of the value from a closer time point. We also utilize the hidden states in the decoder to build connection between history values and current ones. Thus, the proposed model can extract the changing trend of a longer history time span while coping with abrupt changes within a shorter time span. A set of experiments demonstrate that the A&A learning approach can obtain the changing trend of air pollutants, like PM2.5, during a longer time span of 12, 24, or even 48 hours. The approach is also tested under different pollutant concentrations and different periods and the results validate its robustness and generality.

show abstract

Section: Eory-based Methodsmentioning

confidence: 99%

Longer Time Span Air Pollution Prediction: The Attention and Autoencoder Hybrid Learning Model

Zhang

Jin

Zou

et al. 2021

Mathematical Problems in Engineering

View full text Add to dashboard Cite

show abstract

“…We implemented a module system within EarthCube CyberConnector [17,78,79] to realize the proposed mode (Figure 8). The implementation included the searching server system and the client system.…”

Section: Methodsmentioning

confidence: 99%

Advanced Cyberinfrastructure to Enable Search of Big Climate Datasets in THREDDS

Gaigalas

Sun

2019

IJGI

View full text Add to dashboard Cite

Understanding the past, present, and changing behavior of the climate requires close collaboration of a large number of researchers from many scientific domains. At present, the necessary interdisciplinary collaboration is greatly limited by the difficulties in discovering, sharing, and integrating climatic data due to the tremendously increasing data size. This paper discusses the methods and techniques for solving the inter-related problems encountered when transmitting, processing, and serving metadata for heterogeneous Earth System Observation and Modeling (ESOM) data. A cyberinfrastructure-based solution is proposed to enable effective cataloging and two-step search on big climatic datasets by leveraging state-of-the-art web service technologies and crawling the existing data centers. To validate its feasibility, the big dataset served by UCAR THREDDS Data Server (TDS), which provides Petabyte-level ESOM data and updates hundreds of terabytes of data every day, is used as the case study dataset. A complete workflow is designed to analyze the metadata structure in TDS and create an index for data parameters. A simplified registration model which defines constant information, delimits secondary information, and exploits spatial and temporal coherence in metadata is constructed. The model derives a sampling strategy for a high-performance concurrent web crawler bot which is used to mirror the essential metadata of the big data archive without overwhelming network and computing resources. The metadata model, crawler, and standard-compliant catalog service form an incremental search cyberinfrastructure, allowing scientists to search the big climatic datasets in near real-time. The proposed approach has been tested on UCAR TDS and the results prove that it achieves its design goal by at least boosting the crawling speed by 10 times and reducing the redundant metadata from 1.85 gigabytes to 2.2 megabytes, which is a significant breakthrough for making the current most non-searchable climate data servers searchable.

show abstract

“…There are several tutorial Ag-Net Jupyter notebooks (https://github.com/ESIPFed/Ag-Net). We leveraged cloud computing (GeoBrain Cloud) [110][111][112] and parallel processing (NVIDIA CUDA GPU) [93,113] to meet the challenge of processing tremendous number of pixels in remote sensing images (a single Landsat 8 scene contains more than 34 million pixels) [114]. The classified maps will help agriculture monitoring, prediction, and decision making [107,115].…”

Section: Use Case: Ai-based Agriculture Mappingmentioning

confidence: 99%

Geoweaver: Advanced Cyberinfrastructure for Managing Hybrid Geoscientific AI Workflows

Sun

Burgess

et al. 2020

IJGI

Self Cite

View full text Add to dashboard Cite

AI (artificial intelligence)-based analysis of geospatial data has gained a lot of attention. Geospatial datasets are multi-dimensional; have spatiotemporal context; exist in disparate formats; and require sophisticated AI workflows that include not only the AI algorithm training and testing, but also data preprocessing and result post-processing. This complexity poses a huge challenge when it comes to full-stack AI workflow management, as researchers often use an assortment of time-intensive manual operations to manage their projects. However, none of the existing workflow management software provides a satisfying solution on hybrid resources, full file access, data flow, code control, and provenance. This paper introduces a new system named Geoweaver to improve the efficiency of full-stack AI workflow management. It supports linking all the preprocessing, AI training and testing, and post-processing steps into a single automated workflow. To demonstrate its utility, we present a use case in which Geoweaver manages end-to-end deep learning for in-time crop mapping using Landsat data. We show how Geoweaver effectively removes the tedium of managing various scripts, code, libraries, Jupyter Notebooks, datasets, servers, and platforms, greatly reducing the time, cost, and effort researchers must spend on such AI-based workflows. The concepts demonstrated through Geoweaver serve as an important building block in the future of cyberinfrastructure for AI research.

show abstract

Exploring cloud-based Web Processing Service: A case study on the implementation of CMAQ as a Service

Cited by 32 publications

References 29 publications

Longer Time Span Air Pollution Prediction: The Attention and Autoencoder Hybrid Learning Model

Longer Time Span Air Pollution Prediction: The Attention and Autoencoder Hybrid Learning Model

Advanced Cyberinfrastructure to Enable Search of Big Climate Datasets in THREDDS

Geoweaver: Advanced Cyberinfrastructure for Managing Hybrid Geoscientific AI Workflows

Contact Info

Product

Resources

About