Industrial Transfer Learning for Multivariate Time Series Segmentation: A Case Study on Hydraulic Pump Testing Cycles

Abstract

Industrial data scarcity is one of the largest factors holding back the widespread use of machine learning in manufacturing. To overcome this problem, the concept of transfer learning was developed and it achieved high attention in recent industrial research. Our paper focuses on the problem of time series segmentation and presents the first in-depth research about transfer learning for deep-learning based time series segmentation on the example of industrial end-of-line pump testing. In particularly, we investigate if the performance of deep learning models can be increased by pretraining the network with data from other domains. Three different scenarios are analyzed: source and target data being closely related, source and target data being distantly related, and source and target data being non-related. The results demonstrate that transfer learning can enhance the performance of time series segmentation models in respect to accuracy and training speed. The benefit is most clearly seen in scenarios where source and training data are closely related and the number of target training data samples is lowest. However, in the scenario of non-related datasets, cases of negative Transfer Learning were observed as well. Thus, the research emphasizes the potential, but also the challenges of industrial Transfer Learning.

Industrial Transfer Learning for Multivariate Time Series Segmentation: A Case Study on Hydraulic Pump Testing Cycles

“…Almost none of the previous works in the TL field focused on the task of TSS. While TSS problems might be less prevalent in the everyday life of companies and researchers, they still deserve attention due to their importance for fields like human action recognition [ 26 ], sleep staging [ 5 ], and operational state detection [ 6 ]. One contribution had a first look at the topic [ 9 ] and successfully showed that feature extraction process of a TSS–CNN model can be improved when pretraining the model with a large, domain-independent source dataset .…”

“…There is a lack of research on TL strategies for TSS in general, and even more so for TSS use cases in industrial settings. The pump testing dataset we use in this paper was published in the context of one of our previous works [ 6 ]. In that work, the dataset was used for TSS experiments in which the availability of a sufficient amount of training data was ensured.…”

“…The network architecture we use to analyze the impact of TL in this work is called PrecTime. The PrecTime architecture was introduced in 2022 in one of our previous works [ 6 ]. It is designed for multivariate TSS and was already successfully applied to the HPEoL dataset and Opportunity dataset in isolation.…”

“…PrecTime is a time window-based architecture consisting of three modules, i.e., a CNN-based intra-window feature extractor, an LSTM-based inter-window context detection, and a CNN-based intra-window prediction refinement model. The concrete details about the architecture, implementation, loss functions, and hyperparameters are described in [ 6 ]. They were reused in this paper.…”

“…The most successful multivariate TSS approaches proposed in recent years are based on deep learning and use a supervised learning concept (e.g., [ 5 ]). In recent work, we analyzed a TSS use case in manufacturing and proposed a novel deep learning architecture [ 6 ]. The presented network was successful in splitting up the end-of-line testing cycle data of hydraulic pumps into different testing phases based on nine sensor channels.…”