Development of NCL equivalent serial and parallel python routines for meteorological data analysis

Gharat, Jatin; Kumar, Bipin; Ragha, Leena; Barve, Amit; Jeelani, Shaik Mohammad; Clyne, John

doi:10.1177/10943420221077110

Cited by 5 publications

(3 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The NCEP/NCAR raw data is preprocessed through spatial interpolation, wherein the CanESM5 variables are mapped onto the global Gaussian grid T42. To enable the transition from a static grid to a Gaussian grid during the interpolation process, the f2gsh_Wrap interpolation function, created using the NCAR Command Language (NCL), is utilized (Gharat et al, 2022). The CanESM5 parameters on the T42 network projection are displayed in a grid consisting of 128*64 cells for SSP1-1.9, SSP5-8.5, and NCEP data (Swart et al, 2019).…”

Section: Preprocessing Of the Climatic Datamentioning

confidence: 99%

Impacts of climate change on thermal stratification and water quality of Reservoirs

Najari,

Heidarzadeh

2024

Preprint

View full text Add to dashboard Cite

This study examines the impact of climate change on the thermal structure, dissolved oxygen (DO), and phosphate (PO4) values of the Karun IV reservoir, Iran. It covers the baseline period (1995-2014) and future projections (2081-2100). Future meteorological variables were derived from the Canadian Earth System Model version 5 (CanESM5) under two emission scenarios downscaled using a machine-learning approach. The continuous daily inflow of the reservoir over an extended period was simulated by employing the IHACRES rainfall-runoff model, in conjunction with the Water Evaluation and Planning system (WEAP) to replicate water level dynamics and outflow. The evaluation of water quality for the years 2019-2020 was conducted utilizing a calibrated CE-QUAL-W2, and any missing data was approximated through empirical relationships. Our findings indicate that the average surface water temperature and the average PO4 concentration, which respectively stands at 22.8 °C and 0.27 mg/L in the baseline scenario, are projected to increase under both SSP1-1.9 (24.4 °C and 0.31 mg/L) and SSP5-8.5 (26 °C and 0.48 mg/L) scenarios. Whereas the average DO concentration of 8.2 mg/L in the baseline scenario is expected to decline in the mentioned scenarios (7.3 and 6.01 mg/L, respectively). The increase in the temperature of the water and the subsequent decline in DO and PO4 levels in deeper waters pose a significant threat to the sustainable growth of aquatic species in the Karun VI reservoir in the future.

show abstract

Section: Preprocessing Of the Climatic Datamentioning

confidence: 99%

Impacts of climate change on thermal stratification and water quality of Reservoirs

Najari,

Heidarzadeh

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Examples of using Dask to distribute computations can be found in Earth and Climate sciences [13,14]. Also in those fields datasets contain many years of data and can reach sizes of multiple TBs, with non-trivial multidimensional schemas similar to what can be found in HEP.…”

Section: Related Workmentioning

confidence: 99%

Leveraging state-of-the-art engines for large-scale data analysis in High Energy Physics

Padulano

Kabadzhov

Saavedra

et al. 2022

Preprint

View full text Add to dashboard Cite

The Large Hadron Collider (LHC) at CERN has generated a vast amount of information from physics events, reaching peaks of TB of data per day which are then sent to large storage facilities. Traditionally, data processing workflows in the High Energy Physics (HEP) field have leveraged grid computing resources. In this context, users have been responsible for manually parallelising the analysis, sending tasks to computing nodes and aggregating the partial results. Analysis environments in this field have had a common building block in the ROOT software framework. This is the de facto standard tool for storing, processing and visualising HEP data. ROOT offers a modern analysis tool called RDataFrame, which can parallelise computations from a single machine to a distributed cluster while hiding most of the scheduling and result aggregation complexity from users. This is currently done by leveraging Apache Spark as the distributed execution engine, but other alternatives are being explored by HEP research groups. Notably, Dask has rapidly gained popularity thanks to its ability to interface with batch queuing systems, widespread in HEP grid computing facilities. Furthermore, future upgrades of the LHC are expected to bring a dramatic increase in data volumes. This paper presents a novel implementation of the Dask backend for the distributed RDataFrame tool in order to address the aforementioned future trends. The scalability of the tool with both the new backend and the already available Spark backend is demonstrated for the first time on more than two thousand cores, testing a real HEP analysis.

show abstract

“…Around 2015, the Dask project was started, giving the possibility to parallelise the existing Python data science software packages in a familiar way for users [76]. Examples of using Dask to distribute computations can be found in Earth and Climate sciences [108,109]. Also in those fields datasets contain many years of data and can reach sizes of multiple TBs, with nontrivial multidimensional schemas similar to what can be found in HEP.…”

Section: State Of the Artmentioning

confidence: 99%

Distributed Computing Solutions for High Energy Physics Interactive Data Analysis

Padulano¹

View full text Add to dashboard Cite

The scientific research in High Energy Physics (HEP) is characterised by complex computational challenges, which over the decades had to be addressed by researching computing techniques in parallel to the advances in understanding physics. One of the main actors in the field, CERN, hosts both the Large Hadron Collider (LHC) and thousands of researchers yearly who are devoted to collecting and processing the huge amounts of data generated by the particle accelerator. This has historically provided a fertile ground for distributed computing techniques, which led to the creation of the Worldwide LHC Computing Grid (WLCG), a global network providing large computing power for all the experiments revolving around the LHC and the HEP field. Data generated by the LHC so far has already posed challenges for computing and storage. vii de recerca sobre aquesta proposta. Des del punt de vista de l'usuari, això es detalla en forma duna nova interfície que es pot executar en un ordinador portàtil o en milers de nodes informàtics, sense canvis en l'aplicació de l'usuari. Aquest desenvolupament obre la porta a l'explotació de recursos distribuïts a través de motors d'execució estàndard de la indústria que poden escalar a múltiples nodes en clústers HPC o HTC, o fins i tot en ofertes serverless de núvols comercials. Atès que sovint l'anàlisi de dades en aquest camp està limitada per E/S, cal comprendre quins són els possibles mecanismes d'emmagatzematge en memòria cau. En aquest sentit, es va investigar un nou sistema d'emmagatzematge basat en la tecnologia d'emmagatzematge d'objectes com a objectiu per a la memòria cau.En conclusió, el futur de l'anàlisi de dades a HEP presenta reptes des de diverses perspectives, des de l'explotació de recursos informàtics i d'emmagatzematge distribuïts fins al disseny d'interfícies d'usuari ergonòmiques. Els marcs de programari han d'apuntar a l'eficiència i la facilitat d'ús, desvinculant la definició dels càlculs físics dels detalls d'implementació de la seva execució. Aquesta tesi s'emmarca en l'esforç col•lectiu de la comunitat HEP cap a aquests objectius, definint problemes i possibles solucions que poden ser adoptades per futurs investigadors. viii

show abstract

Development of NCL equivalent serial and parallel python routines for meteorological data analysis

Cited by 5 publications

References 16 publications

Impacts of climate change on thermal stratification and water quality of Reservoirs

Impacts of climate change on thermal stratification and water quality of Reservoirs

Leveraging state-of-the-art engines for large-scale data analysis in High Energy Physics

Distributed Computing Solutions for High Energy Physics Interactive Data Analysis

Contact Info

Product

Resources

About