Shaking the foundations: delusions in sequence models for interaction and control

Abstract

The recent phenomenal success of language models has reinvigorated machine learning research, and large sequence models such as transformers are being applied to a variety of domains. One important problem class that has remained relatively elusive however is purposeful adaptive behavior. Currently there is a common perception that sequence models "lack the understanding of the cause and effect of their actions" leading them to draw incorrect inferences due to auto-suggestive delusions. In this report we explain where this mismatch originates, and show that it can be resolved by treating actions as causal interventions. Finally, we show that in supervised learning, one can teach a system to condition or intervene on data by training with factual and counterfactual error signals respectively.

A Generalist Agent

“…Generating actions using an autoregressive model can lead to causal "self-delusion" biases when there are confounding variables (Ortega et al, 2021). For example, sampling actions can condition the model to solve the wrong task when multiple tasks share similar observation and actions specifications.…”

Language Model Cascades

“…The Eliciting Latent Knowledge proposal (ELK, 2022) suggests making latent variables explicit, modelled using a Bayesian network, to improve interpretability and safety for advanced AI systems. Ortega et al (2021) explains a formalism for LM finetuning with causal graphical models in order to extend the pre-dictive capabilities of AI agents towards more adaptive behaviour. They focus on analysing an auto-regressive action (random variable) prediction scheme in the interactive setting of RL where a model is simultaneously a generator and predictor of data.…”

Learning Ergonomic Control in Human–Robot Symbiotic Walking

“…However, when applying this same model to the MPC process to optimize the control trajectory for specific results we may not want to optimize future control states based on past executed controls. Recent theoretical insights [44] into learning models for control show that conditioning on prior actions can cause "self-delusions," i.e., the model takes its own actions as evidence about the world, thereby slowly corrupting the inference process. To avoid the occurence of such self-delusions, we zero out all weights below the diagonal of the weight matrix w r|u from (15), which represent the delusional effects of future controls on past actions.…”