Autonomous recovery from hostile code insertion using distributed reflection

Abstract

In a hostile environment, an autonomous cognitive system requires a reflective capability to detect problems in its own operation and recover from them without external intervention. We present an architecture in which reflection is distributed so that components mutually observe and protect each other, and where the system has a distributed model of all its components, including those concerned with the reflection itself. Some reflective (or "meta-level") components enable the system to monitor its execution traces and detect anomalies by comparing them with a model of normal activity. Other components monitor "quality" of performance in the application domain. Implementation in a simple virtual world shows that the system can recover from certain kinds of hostile code attacks that cause it to make wrong decisions in its application domain, even if some of its self-monitoring components are also disabled.

The well-designed young mathematician

“…are not discussed here: that they represent structures, events, processes and generalisations about what is going on in the machine is all I am concerned with. Compare McCarthy in [23] and Kennedy in [20]: although neither says so, they both assume that the contents of self-observation are not the physical contents of the machine, but contents of virtual machines implemented in the physical machinery.…”

“…stop when one of the processes cannot continue (no more marbles) or stop when some stage in one of the processes has been reached (fetching six marbles), as described in [41,Chapter 8]. 20 Fourth, it became clear that some discoveries made by a learner could start off empirical (e.g. discovering that counting a row of objects from left to right gave the same result as counting from right to left) but as the learner acquired a deeper understanding of what was going on such discoveries could be seen not to be empirical, but examples of necessary truths.…”