Transformer enhanced by local perception self-attention for dynamic soft sensor modeling of industrial processes

Abstract

The use of dynamic soft sensor modeling methods to mine the time-varying and dynamic characteristics of industrial process data is of great significance for improving production efficiency and quality, given the rapid development of industrial processes and the increasing prominence of dynamic changes in the production process. However, existing dynamic soft sensor methods have limited long-term memory capacity, making it difficult to capture the long dynamic dependence, which can severely affect the results of the soft sensor model. To address this issue, we propose a dynamic soft sensor model based on local perception transformer (LPT), where the transformer is applied to achieve global perception of the variables. Through the application of the self-attention mechanism in the transformer encoder, the dynamic tracking and prediction of parameters can be realized by assigning different weights to the process variables and quality variables at different time steps, thereby adapting to the time-varying nature of the process. Additionally, convolution is used to generate a Query and Key in the self-attention mechanism, thereby enhancing local information learning.
The proposed dot product self-attention calculation method effectively utilizes local information, thereby reducing the potential impact of abnormal data at a certain moment.Furthermore, by utilizing LSTM to extract time series information, the final prediction result was obtained. In soft sensor modeling experiments of the sulfur recovery unit (SRU) and debutanizer tower, our proposed model demonstrated higher prediction accuracy compared to other methods, such as SVR, MLP, LSTM, CNN+LSTM, and Vanilla Transformer.