Multilevel Sensor Fusion With Deep Learning

Vielzeuf, Valentin; Lechervy, Alexis; Pateux, Stéphane; Jurie, Frédéric

doi:10.1109/lsens.2018.2878908

Cited by 14 publications

(7 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The combination of information from different sensors at the data level is referred to as early information fusion of multimodal data [28]. The use of early fusion with several multimodal datasets has been discussed in [29] and followed in [30] in the implementation of a fault detection system in rotatory machines.…”

Section: Sensory Data Preprocessingmentioning

confidence: 99%

Sensitivity Analysis of Sensors in a Hydraulic Condition Monitoring System Using CNN Models

König

Helmi

2020

Sensors

View full text Add to dashboard Cite

Condition monitoring (CM) is a useful application in industry 4.0, where the machine’s health is controlled by computational intelligence methods. Data-driven models, especially from the field of deep learning, are efficient solutions for the analysis of time series sensor data due to their ability to recognize patterns in high dimensional data and to track the temporal evolution of the signal. Despite the excellent performance of deep learning models in many applications, additional requirements regarding the interpretability of machine learning models are getting relevant. In this work, we present a study on the sensitivity of sensors in a deep learning based CM system providing high-level information about the relevance of the sensors. Several convolutional neural networks (CNN) have been constructed from a multisensory dataset for the prediction of different degradation states in a hydraulic system. An attribution analysis of the input features provided insights about the contribution of each sensor in the prediction of the classifier. Relevant sensors were identified, and CNN models built on the selected sensors resulted equal in prediction quality to the original models. The information about the relevance of sensors is useful for the system’s design to decide timely on the required sensors.

show abstract

Section: Sensory Data Preprocessingmentioning

confidence: 99%

Sensitivity Analysis of Sensors in a Hydraulic Condition Monitoring System Using CNN Models

König

Helmi

2020

Sensors

View full text Add to dashboard Cite

show abstract

“…The DL models for sensory integration usually process different modalities separately in different data streams, until sufficiently high-level features are extracted to fuse the information [32], [39], [42], [51]. The best approach to choose the abstraction level (a layer of the DL model) at which the information is fused is application specific, so it may as well be learned with an appropriate DL model [52].…”

Section: A Perceptionmentioning

confidence: 99%

Deep Learning in Robotics: Survey on Model Structures and Training Strategies

Karoly

Galambos

Kuti

et al. 2021

IEEE Trans. Syst. Man Cybern, Syst.

View full text Add to dashboard Cite

The ever-increasing complexity of robot applications induces the need for methods to approach problems with no (viable) analytical solution. Deep learning (DL) provides a set of tools to address this kind of problems. This survey presents a categorization of the major challenges in robotics that leverage DL technologies and introduces representative examples of successful solutions for the described problems. We also consider the question when and whether to use modular, monolithic models or end-to-end DL, in order to provide a guideline for the selection of the correct model structure and training strategy. By doing so, the current role and adaptability of different techniques at different hierarchical levels of a robot-application can be highlighted, thus providing a well-structured basis to assist future approaches. Index Terms-Deep learning (DL), machine learning (ML), manipulators, mobile robots, neural networks, robot control, robot learning. I. INTRODUCTION C OMPUTERS can easily solve formal problems that are demanding for humans. However, the increasing need for adaptive systems requires the solution of tasks that are hard to formulate, but can be easily solved by humans, such as the recognition and manipulation of objects. In order to perform such tasks, a certain complex knowledge of the environment is inevitable. The automatic extraction of the required knowledge is called machine learning (ML). The way the data is presented to the ML system, heavily influences how well the extracted knowledge represents the given problem. The ML approaches that also perform feature extraction, using multiple hierarchical artificial neural network layers, are referred to as deep learning (DL) [1], [2]. The theoretical background of DL had been introduced for a long time, when it finally gained widespread popularity, i.a., thanks to the winner entry of the ImageNet Challenge Manuscript

show abstract

“…Combining sensor inputs within a single deep network is a less commonly tackled problem. Sensory combinations have been studied using late convolutional fusion followed by fullyconnected layers [31], using convolutional LSTMs on top of sensory-independent representations [44], or through multiobjective regularization on different embedding combinations for sensory-specific networks [54]. Such approaches focus on fusing low-dimensional sensors, ignoring high-dimensional signals such as videos.…”

Section: Sensor-based Activity Recognitionmentioning

confidence: 99%

Heterogeneous Non-Local Fusion for Multimodal Activity Recognition

Byvshev

Mettes

Xiao

2020

Proceedings of the 2020 International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

In this work, we investigate activity recognition using multimodal inputs from heterogeneous sensors. Activity recognition is commonly tackled from a single-modal perspective using videos. In case multiple signals are used, they come from the same homogeneous modality, e.g. in the case of color and optical flow. Here, we propose an activity network that fuses multimodal inputs coming from completely different and heterogeneous sensors. We frame such a heterogeneous fusion as a non-local operation. The observation is that in a non-local operation, only the channel dimensions need to match. In the network, heterogeneous inputs are fused, while maintaining the shapes and dimensionalities that fit each input. We outline both asymmetric fusion, where one modality serves to enforce the other, and symmetric fusion variants. To further promote research into multimodal activity recognition, we introduce GloVid, a first-person activity dataset captured with video recordings and smart glove sensor readings. Experiments on GloVid show the potential of heterogeneous non-local fusion for activity recognition, outperforming individual modalities and standard fusion techniques. CCS CONCEPTS• Computing methodologies → Activity recognition and understanding.

show abstract

Multilevel Sensor Fusion With Deep Learning

Cited by 14 publications

References 16 publications

Sensitivity Analysis of Sensors in a Hydraulic Condition Monitoring System Using CNN Models

Sensitivity Analysis of Sensors in a Hydraulic Condition Monitoring System Using CNN Models

Deep Learning in Robotics: Survey on Model Structures and Training Strategies

Heterogeneous Non-Local Fusion for Multimodal Activity Recognition

Contact Info

Product

Resources

About