Runtime Monitoring Neuron Activation Patterns

Cheng, Chih-Hong; Nührenberg, Georg; Yasuoka, Hirotoshi

doi:10.23919/date.2019.8714971

Cited by 59 publications

(64 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…During deployment, run-time monitoring is performed by comparing the current values in the layers with the abstraction. Another related work is proposed by Cheng et al [37]. They stored the neuron activation pattern in an abstract form and used Hamming distance to compare the generated pattern at run-time to the abstract form.…”

Section: B Monitoring Based On Inconsistencies During Inferencementioning

confidence: 99%

Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends

2021

View full text Add to dashboard Cite

As deep learning continues to dominate all state-of-the-art computer vision tasks, it is increasingly becoming an essential building block for robotic perception. This raises important questions concerning the safety and reliability of learning-based perception systems. There is an established field that studies safety certification and convergence guarantees of complex software systems at design-time. However, the unknown future deployment environments of an autonomous system and the complexity of learning-based perception make the generalization of design-time verification to run-time problematic. In the face of this challenge, more attention is starting to focus on run-time monitoring of performance and reliability of perception systems with several trends emerging in the literature in the face of this challenge. This paper attempts to identify these trends and summarise the various approaches to the topic.

show abstract

Section: B Monitoring Based On Inconsistencies During Inferencementioning

confidence: 99%

Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends

2021

View full text Add to dashboard Cite

show abstract

“…There are a few existing methods on abstraction of deep learning. For example, in [4], a Boolean abstraction on the ReLU activation pattern of some specific layer is considered and monitored. Conversely of Boolean abstraction, [7] consider box abstractions.…”

Section: Runtime Monitoringmentioning

confidence: 99%

Safety and reliability of deep learning

Huang

2021

Proceedings of the 1st International Workshop on Verification of Autonomous &Amp; Robotic Systems

View full text Add to dashboard Cite

Robotics and Autonomous Systems (RAS) become ever more relying on deep learning components to support their perception and decision making. Given RAS will inevitably be applied to safety critical applications, efforts are needed to ensure that the deep learning is safe and reliable. In this lecture, I will give a brief overview on recent progress in the verification and validation techniques for deep learning, focusing on two major safety and reliability risks, i.e., robustness and generalisation. We consider formal verification, statistical evaluation, reliability assessment, and runtime monitoring techniques, all of which complement with each other in providing assurance to the reliability of deep learning in operation. The challenges and future directions will also be discussed.

show abstract

“…Although it does not incorporate the specification of test data, i.e., requirements specification, runtime monitoring of neuron activation patterns is an approach to detect change points [14]. It creates a monitor of neuron activation patterns after training time, and runs the monitor at operation time to measure the deviation from training time.…”

Section: Related Work and Research Directions For Requirements Of Mamentioning

confidence: 99%

Engineering problems in machine learning systems

Kuwajima

Yasuoka²,

Nakae³

2020

Mach Learn

Self Cite

View full text Add to dashboard Cite

Fatal accidents are a major issue hindering the wide acceptance of safety-critical systems that employ machine learning and deep learning models, such as automated driving vehicles. In order to use machine learning in a safety-critical system, it is necessary to demonstrate the safety and security of the system through engineering processes. However, thus far, no such widely accepted engineering concepts or frameworks have been established for these systems. The key to using a machine learning model in a deductively engineered system is decomposing the data-driven training of machine learning models into requirement, design, and verification, particularly for machine learning models used in safety-critical systems. Simultaneously, open problems and relevant technical fields are not organized in a manner that enables researchers to select a theme and work on it. In this study, we identify, classify, and explore the open problems in engineering (safety-critical) machine learning systems -that is, in terms of requirement, design, and verification of machine learning models and systems -as well as discuss related works and research directions, using automated driving vehicles as an example. Our results show that machine learning models are characterized by a lack of requirements specification, lack of design specification, lack of interpretability, and lack of robustness. We also perform a gap analysis on a conventional system quality standard SQuARE with the characteristics of machine learning models to study quality models for machine learning systems. We find that a lack of requirements specification and lack of robustness have the greatest impact on conventional quality models.Preprint. Work in progress.

show abstract

Runtime Monitoring Neuron Activation Patterns

Cited by 59 publications

References 20 publications

Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends

Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends

Safety and reliability of deep learning

Engineering problems in machine learning systems

Contact Info

Product

Resources

About