Cloud-Native Repositories for Big Scientific Data

We use data from a realistic high-resolution coupled climate model to train a neu-6 ral network (NN) and perform offline tests 7 • We learn a stochastic parameterization of subgrid momentum forcing with the NN 8 that generalizes well to different regimes and climates 9 • The stochastic parameterization results in a stable implementation in an ideal-10 ized ocean model that improves the kinetic energy of the flow 11

show abstract

Section: Data For Training and Validationmentioning

confidence: 99%

Stochastic‐Deep Learning Parameterization of Ocean Momentum Forcing

Guillaumin

Zanna

2021

J Adv Model Earth Syst

View full text Add to dashboard Cite

show abstract

“…The nominal horizontal resolution of the ocean component of CM2.6 is 1/10 • , therefore resolving mesoscale eddies in many regions of the ocean (Hallberg, 2013). The data and tools for analysis were obtained from the Pangeo platform (Abernathey et al, 2021). The data used in the present work consists of the high-resolution simulated ocean surface velocity field u with components u (zonal) and v (meridional).…”

Section: Data For Training and Validationmentioning

confidence: 99%

Stochastic Deep Learning parameterization of Ocean Momentum Forcing

Guillaumin

Zanna

2021

Preprint

View full text Add to dashboard Cite

Coupled climate simulations that span several hundred years cannot be run at a high-enough spatial resolution to resolve mesoscale ocean dynamics. These mesoscale dynamics backscatter to macroscales. Recently, several studies have considered Deep Learning to parameterize subgrid forcing within macroscale ocean equations using data from idealized simulations. In this manuscript, we present a stochastic Deep Learning parameterization that is trained on data generated by CM2.6, a highresolution state-of-the-art coupled climate model with nominal resolution 1/10°. We train a Convolutional Neural Network for the subgrid momentum forcing using macroscale surface velocities from a few selected subdomains. At each location and each time step of the coarse grid, rather than predicting a single number, we predict the mean and standard deviation of a Gaussian probability distribution. This approach requires training our neural network to minimize a negative log-likelihood loss function rather than the Mean Square Error, which has been the standard in applications of Deep Learning to the problem of parameterizations. Each prediction of the mean subgrid forcing can be associated with an uncertainty estimate and can form the basis for a stochastic subgrid parameterization. Offline tests show that our parameterization generalizes well to the global oceans, and a climate with increased CO2 levels, without further training. We test our stochastic parameterization in an idealized shallow water model. The implementation is stable and improves some statistics of the flow. Our work demonstrates the potential of combining Deep Learning tools with a probabilistic approach in parameterizing unresolved ocean dynamics.

show abstract

“…Zarr 31,32 is a relatively new data storage format that is flexible, compressible, and designed to be accessed with open-source software using cloud or local computing resources. The work in Reference 33 presented webGlobe, a browser-based GIS framework for interacting with climate data and other datasets available in a similar format, with optimization facilities and AI capacities for data formats based on NetCDF.…”

mentioning

confidence: 99%

A novel approach for large‐scale environmental data partitioning on cloud and on‐premises storage for compute continuum applications

Mellone,

De Vita,

Sánchez‐Gallegos

et al. 2023

Concurrency and Computation

View full text Add to dashboard Cite

SummaryCloud‐based services have proved useful in several research fields, such as engineering, health science, and astrophysics, to mention a few examples. The computational environmental science community developed a strong need for cloud facilities to store, process, and manage data from observations and numerical models for simulations and forecasts. Weather forecast models and global sensor networks deal with multidimensional geo‐referenced data∖sets. However, environmental data consumer applications usually require a relatively small amount of multidimensional input data slice to analyze a specific area or time interval. Hence, reducing data dimension for information retrieval is mandatory. This paper presents a twofold solution: a technique to load and retrieve the sliced multidimensional data set on different cloud services such as Amazon Web Service (AWS), Google Cloud Platform, and Microsoft Azure. The experimental results performed on these cloud services highlight that the proposed method can significantly speed up the process of loading and retrieving the data slices compared to working with the entire data set in bulk or OPeNDAP server.

show abstract

Cloud-Native Repositories for Big Scientific Data

Cited by 4 publications

References 0 publications

Stochastic‐Deep Learning Parameterization of Ocean Momentum Forcing

Stochastic‐Deep Learning Parameterization of Ocean Momentum Forcing

Stochastic Deep Learning parameterization of Ocean Momentum Forcing

A novel approach for large‐scale environmental data partitioning on cloud and on‐premises storage for compute continuum applications

Contact Info

Product

Resources

About