Application of Recurrent Neural Networks to Rainfall-runoff Processes

Pan, Tsung-Yi; Wang, Ru-yih; Lai, Jihn‐Sung; Yu, Hwa-lung

doi:10.5772/5542

Cited by 2 publications

(2 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Static positional encoding calculates gradients by accumulation and is only related to sequence length and word vector standard deviation. RNNs calculate gradients by continuous multiplication at almost all positions during back propagation and the gradients are close to 0 [17], it may face the problem of vanishing gradient. For one transformer block, the standard deviation of word vector is usually of the order of 0.02 [4], and for other blocks, LayerNormalization ensures that the word vector standard deviation is of the order of 1 [17].…”

Section: Cumsum Calculationmentioning

confidence: 99%

“…RNNs calculate gradients by continuous multiplication at almost all positions during back propagation and the gradients are close to 0 [17], it may face the problem of vanishing gradient. For one transformer block, the standard deviation of word vector is usually of the order of 0.02 [4], and for other blocks, LayerNormalization ensures that the word vector standard deviation is of the order of 1 [17]. The position parameter of cumsum calculation can always maintain a reasonable gradient.…”

Section: Cumsum Calculationmentioning

confidence: 99%

See 1 more Smart Citation

Sumformer: recursive positional encoding for transformer in short text classification

Zhan

Huang

et al. 2023

3rd International Conference on Artificial Intelligence, Automation, and High-Performance Computing (AIAHPC 2023)

View full text Add to dashboard Cite

In various transformers, positional encoding is used to compensate for the inability of the attention mechanism to capture positional information between words. Previous research on transformers' temporal modeling has utilized recursive and relative positional encoding based on the Recurrent Neural Network (RNN). Recursive positional encoding captures linear text structure but lacks parallelization, hindering speed. In contrast, relative positional encoding ignores linear text structure, leading to weaker performance in short text classification compared to recursive positional encoding. To address the issues, we propose a model, sumformer, which mainly includes two parts different from the other transformers: cumsum calculation and summer initialization. Cumsum calculation simplifies the feature extraction part of RNN by a substitution method, replacing the dynamic rate function of RNNs with static trainable position parameters, and preserves the recursive structure, which enables the model to capture the linear structure information of the text through cumsum calculation method and maintains a low time overhead compared to RNNs. In addition, the summer initialization method, which limits the highest standard deviation of the positional parameter, enables the model to pay attention to the multi-level information of the text during initialization, with the richer optimization space, thereby improving the convergence ability of the model. The experimental results show the sumformer achieves roughly a 3% improvement in performance and a 58% improvement in speed compared to existing transformers based on recursive positional encoding. It achieves better short text classification faster, and summer initialization also can improve the performance without increasing training and inference time.

show abstract

Section: Cumsum Calculationmentioning

confidence: 99%

Section: Cumsum Calculationmentioning

confidence: 99%

Sumformer: recursive positional encoding for transformer in short text classification

Zhan

Huang

et al. 2023

3rd International Conference on Artificial Intelligence, Automation, and High-Performance Computing (AIAHPC 2023)

View full text Add to dashboard Cite

show abstract

Hybrid neural networks in rainfall-inundation forecasting based on a synthetic potential inundation database

Pan

Lai

Chang

et al. 2011

Nat. Hazards Earth Syst. Sci.

View full text Add to dashboard Cite

Abstract. This study attempts to achieve real-time rainfallinundation forecasting in lowland regions, based on a synthetic potential inundation database. With the principal component analysis and a feed-forward neural network, a rainfall-inundation hybrid neural network (RiHNN) is proposed to forecast 1-h-ahead inundation depth as hydrographs at specific representative locations using spatial rainfall intensities and accumulations. A systematic procedure is presented to construct the RiHNN, which combines the merits of detailed hydraulic modeling in flood-prone lowlands via a two-dimensional overland-flow model and time-saving calculation in a real-time rainfall-inundation forecasting via ANN model. Analytical results from the RiHNNs with various principal components indicate that the RiHNNs with fewer weights can have about the same performance as a feed-forward neural network. The RiHNNs evaluated through four types of real/synthetic rainfall events also show to fit inundation-depth hydrographs well with high rainfall. Moreover, the results of real-time rainfall-inundation forecasting help the emergency manager set operational responses, which are beneficial for flood warning preparations.

show abstract

Application of Recurrent Neural Networks to Rainfall-runoff Processes

Cited by 2 publications

References 29 publications

Sumformer: recursive positional encoding for transformer in short text classification

Sumformer: recursive positional encoding for transformer in short text classification

Hybrid neural networks in rainfall-inundation forecasting based on a synthetic potential inundation database

Contact Info

Product

Resources

About