A multi-head attention-based transformer model for traffic flow forecasting with a comparative analysis to recurrent neural networks

Reza, Selim; Ferreira, Marta Campos; Machado, J.J.M.; Tavares, João Manuel R. S.

doi:10.1016/j.eswa.2022.117275

Cited by 100 publications

(33 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the outcomes of the transformer run in the experimental instance are not indisputably better than those of the LSTM and GRU models. In Selim's research [24] about the traffic flow forecasting method, the MAPE of the LSTM model achieved 12.37%, the MAPE of the GRU model reached 12.66% and the best forecast outcome's MAPE in this study was 21.33%. One of the study's research weaknesses is the dearth of historical passenger flow data, which contributed to the fact that the effect of MAPE in the tests was not very noteworthy.…”

Section: Discussionmentioning

confidence: 50%

“…These researchers have achieved rich results, including long, medium and short term prediction of traffic station flow [20], transportation mode flow [18,21] and traffic networks [19] based on massive amounts of traffic data, based on the widely used LSTM, GRU and other algorithms. Lately, the transformer algorithm has been better used in traffic timeseries prediction [22]; it can train the model by removing the spatiotemporal characteristics of traffic data [23], and also helps with the dependence issue within long series data processing [24].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

LSTM-Based Transformer for Transfer Passenger Flow Forecasting between Transportation Integrated Hubs in Urban Agglomeration

Yue

2023

Applied Sciences

View full text Add to dashboard Cite

A crucial component of multimodal transportation networks and long-distance travel chains is the forecasting of transfer passenger flow between integrated hubs in urban agglomerations, particularly during periods of high passenger flow or unusual weather. Deep learning is better suited to managing massive amounts of traffic data and predicting extended time series. In order to solve the problem of gradient explosion or gradient disappearance that recurrent neural networks are prone to when dealing with long time sequences, this study used a transformer prediction model to estimate short-term transfer passenger flow between two integrated hubs in an urban agglomeration and a long short-term memory network to incorporate previous historical data. The experimental analysis uses two sets of transfer passenger data from the Beijing-Tianjin-Hebei urban agglomeration, collected every 30 min in May 2021 on the transfer corridors between an airport and a high-speed railway station. The findings demonstrate the high adaptability and good performance of the suggested model in passenger flow forecasting. The suggested model and forecasting outcomes assist management in making capacity adjustments in time to correspond with changes, enhance the effectiveness of multimodal transportation systems in urban agglomerations and significantly enhance the service of long-distance multimodal passenger travel.

show abstract

Section: Discussionmentioning

confidence: 50%

Section: Introductionmentioning

confidence: 99%

LSTM-Based Transformer for Transfer Passenger Flow Forecasting between Transportation Integrated Hubs in Urban Agglomeration

Yue

2023

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…A recurrent neural network, in terms of network architecture, keeps track of prior data and applies that data to affect the output of subsequent nodes. In other words, a RNN's hidden layers are interconnected, and their inputs contain both the outputs of the input layers and the outputs of the hidden levels from earlier in time [ 27 ]. The RNN complex can be conceptualized as the outcome of endless replication of the same neural network structure.…”

Section: Models and Evaluation Methodsmentioning

confidence: 99%

Evaluation of the Practical Effects of Environmental Measures in the Conservation of Architectural Heritage in Yan’an Based on Recurrent Neural Networks

Wang

2022

Journal of Environmental and Public Health

View full text Add to dashboard Cite

Yan’an is one of the “two holy places” of the Chinese nation and the Chinese revolution and is one of the first cities of historical and cultural significance and an outstanding tourist city in China, as announced by the state council. The evaluation of the effectiveness of environmental conservation is one of the very important elements of the conservation of Yan’an’s architectural heritage. However, the existing evaluation methods cannot provide new solutions for decision-making, the meaning of the comprehensive evaluation function is unclear, the naming clarity is low, there is less quantitative data and more qualitative components, and the results are not easily convincing. This paper proposes a method for evaluating the practical effects of environmental class measures in the conservation of Yan’an’s architectural heritage based on recurrent neural networks. The recurrent neural network makes full use of the memory function in the network, considers the causal relationship of the actual effect, and efficiently evaluates the existing measures. In comparison with factor analysis and hierarchical analysis, this paper has greater applicability in evaluating the practical effects of environmental measures in the conservation of Yan’an’s architectural heritage and is basically consistent with the results of the theoretical analysis. It provides a scientific basis for the construction and implementation of environmental measures for the architectural heritage of Yan’an.

show abstract

“…Then, the resulting series is re-weighted using a MultiHead Attention layer and added to the previous result. This operation of adding the result of a layer with its input is a common practice known as "residual connections" and is commonly used with convolutional layers for image processing (He et al, 2015) and for time-series processing with Attention layers (Reza et al, 2022;Vaswani et al, 2017). However, it is important to address a key difference between our model and other works with similar architectures: after the MultiHead Attention layers and the residual addition, a Layer Normalization (Ba et al, 2016) operation is usually applied.…”

Section: Deep Neural Network Architecturementioning

confidence: 99%

Neural Networks for Operational SYM‐H Forecasting Using Attention and SWICS Plasma Features

Collado‐Villaverde,

Muñoz,

Cid

2023

Space Weather

View full text Add to dashboard Cite

In this work, we present an Artificial Neural Network for operational forecasting of the SYM‐H geomagnetic index up to 2 hr ahead using the Interplanetary Magnetic Field, the solar wind plasma features and previous SYM‐H values. Former works that forecast the SYM‐H index use data measured by ACE, in particular from the MAG and SWEPAM instruments. However, the plasma data present a high amount of missing samples. This issue has been addressed in the literature, often using linear interpolation, which leads to a non‐accurate data reconstruction, specially when the features are missing during the most intense periods of a geomagnetic storm. To overcome that issue, we use ACE's Solar Wind Ion Composition Spectrometer (SWICS) data to fill the missing plasma features. To validate this technique, we compare the results of our forecasting model trained using plasma features in two ways: only using SWEPAM and performing linear interpolation and using SWICS to fill the missing values in SWEPAM. Then, both models are evaluated in an operational scenario, when only SWEPAM data are available and interpolation can only be performed if the missing values are surrounded by valid measurements. In both cases, our model outperforms the current literature forecasting the SYM‐H one and 2 hr ahead, yielding the best results when the training has been done using the data completed using SWICS measurements.

show abstract

A multi-head attention-based transformer model for traffic flow forecasting with a comparative analysis to recurrent neural networks

Cited by 100 publications

References 29 publications

LSTM-Based Transformer for Transfer Passenger Flow Forecasting between Transportation Integrated Hubs in Urban Agglomeration

LSTM-Based Transformer for Transfer Passenger Flow Forecasting between Transportation Integrated Hubs in Urban Agglomeration

Evaluation of the Practical Effects of Environmental Measures in the Conservation of Architectural Heritage in Yan’an Based on Recurrent Neural Networks

Neural Networks for Operational SYM‐H Forecasting Using Attention and SWICS Plasma Features

Contact Info

Product

Resources

About