Synthetic rainfall data generator development through decentralised model training

Welten, Sascha; Holt, Adrian; Hofmann, Julian; Schelter, Lennart; Klopries, Elena‐Maria; Wintgens, Thomas; Decker, Stefan

doi:10.1016/j.jhydrol.2022.128210

Cited by 5 publications

(3 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, it has the disadvantage of requiring more computational resources and some knowledge of its internals. Compared with artificial neural networks (ANNs) (Welten et al, 2022), point processes may be less flexible in terms of incorporating nonlinear relationships or external information. However, ANNs are still not widely used as rainfall synthetic generators, being more commonly used for rainfall-runoff prediction.…”

Section: Discussionmentioning

confidence: 99%

NEOPRENE v1.0.1: a Python library for generating spatial rainfall based on the Neyman–Scott process

Diez-Sierra,

Navas,

del Jesus

2023

Geosci. Model Dev.

View full text Add to dashboard Cite

Abstract. Long time series of rainfall at different levels of aggregation (daily or hourly in most cases) constitute the basic input for hydrological, hydraulic and climate studies. However, oftentimes the length, completeness, time resolution or spatial coverage of the available records falls short of the minimum requirements to build robust estimations. Here, we introduce NEOPRENE, a Python library to generate synthetic time series of rainfall. NEOPRENE simulates multi-site synthetic rainfall that reproduces observed statistics at different time aggregations. Three case studies exemplify the use of the library, focusing on extreme rainfall, as well as on disaggregating daily rainfall observations into hourly rainfall records. NEOPRENE is distributed from GitHub with an open license (GPLv3), free for research and commercial purposes alike. We also provide Jupyter notebooks with the example use cases to promote its adoption by researchers and practitioners involved in vulnerability, impact and adaptation studies.

show abstract

Section: Discussionmentioning

confidence: 99%

NEOPRENE v1.0.1: a Python library for generating spatial rainfall based on the Neyman–Scott process

Diez-Sierra,

Navas,

del Jesus

2023

Geosci. Model Dev.

View full text Add to dashboard Cite

show abstract

“…• A. GAN on clients: 7 papers [60,61,63,66,69,73,75] had a GAN on each client but not on the server. Usually, each client trains its GAN, sends the parameters to the server, which aggregates them and sends them back to the client, and so on.…”

Section: What Methods Have Been Used For Federated Synthesis?mentioning

confidence: 99%

Federated learning for generating synthetic data: a scoping review

Little,

Elliot,

Allmendinger

2023

IJPDS

View full text Add to dashboard Cite

IntroductionFederated Learning (FL) is a decentralised approach to training statistical models, where training is performed across multiple clients, producing one global model. Since the training data remains with each local client and is not shared or exchanged with other clients the use of FL may reduce privacy and security risks (compared to methods where multiple data sources are pooled) and can also address data access and heterogeneity problems. Synthetic data is artificially generated data that has the same structure and statistical properties as the original but that does not contain any of the original data records, therefore minimising disclosure risk. Using FL to produce synthetic data (which we refer to as "federated synthesis") has the potential to combine data from multiple clients without compromising privacy, allowing access to data that may otherwise be inaccessible in its raw format. ObjectivesThe objective was to review current research and practices for using FL to generate synthetic data and determine the extent to which research has been undertaken, the methods and evaluation practices used, and any research gaps. MethodsA scoping review was conducted to systematically map and describe the published literature on the use of FL to generate synthetic data. Relevant studies were identified through online databases and the findings are described, grouped, and summarised. Information extracted included article characteristics, documenting the type of data that is synthesised, the model architecture and the methods (if any) used to evaluate utility and privacy risk. ResultsA total of 69 articles were included in the scoping review; all were published between 2018 and 2023 with two thirds (46) in 2022. 30% (21) were focussed on synthetic data generation as the main model output (with 6 of these generating tabular data), whereas 59% (41) focussed on data augmentation. Of the 21 performing federated synthesis, all used deep learning methods (predominantly Generative Adversarial Networks) to generate the synthetic data. ConclusionsFederated synthesis is in its early days but shows promise as a method that can construct a global synthetic dataset without sharing any of the local client data. As a field in its infancy there are areas to explore in terms of the privacy risk associated with the various methods proposed, and more generally in how we measure those risks.

show abstract

“…8 In the industrial sciences, synthetic data has been used to generate test burst data to prevent corrosion in metal pipes, 15 simulating electrical load data, 35 government sector traffic volume forecasting, 37 and rainwater synthetics for flood prediction. 32 Most of these models generate continuous values in contrast to our problem of synthetic generation of nominal time-series data.…”

Section: Related Workmentioning

confidence: 99%

Synthetically generating human-like data for sequential decision-making tasks via reward-shaped imitation learning

Brandt

Dasgupta

2023

Synthetic Data for Artificial Intelligence and Machine Learning: Tools, Techniques, and Applications

View full text Add to dashboard Cite

We consider the problem of synthetically generating data that can closely resemble human decisions made in the context of an interactive human-AI system like a computer game. We propose a novel algorithm that can generate synthetic, human-like, decision making data while starting from a very small set of decision making data collected from humans. Our proposed algorithm integrates the concept of reward shaping with an imitation learning algorithm to generate the synthetic data. We have validated our synthetic data generation technique by using the synthetically generated data as a surrogate for human interaction data to solve three sequential decision making tasks of increasing complexity within a small computer game-like setup. Different empirical and statistical analyses of our results show that the synthetically generated data can substitute the human data and perform the game-playing tasks almost indistinguishably, with very low divergence, from a human performing the same tasks.

show abstract

Synthetic rainfall data generator development through decentralised model training

Cited by 5 publications

References 23 publications

NEOPRENE v1.0.1: a Python library for generating spatial rainfall based on the Neyman–Scott process

NEOPRENE v1.0.1: a Python library for generating spatial rainfall based on the Neyman–Scott process

Federated learning for generating synthetic data: a scoping review

Synthetically generating human-like data for sequential decision-making tasks via reward-shaped imitation learning

Contact Info

Product

Resources

About