Information fusion for decision making

Abstract

Making intelligent decisions in autonomous systems often requires the use of all available information. This information generally comes from multiple sources and may be expressed through multiple modalities. This thesis presents a collection of different approaches for processing and fusing sensor data that have been developed for systems that enjoy multiple sensors and/or sensing modalities. Within these approaches we focus on information fusion in the form of emerged communication for team tasks in multi-agent systems in the first part. After that we investigate information fusion in the context of active sensing by treatment of an agent as a hybrid switched system. Then we look at information fusion in a probabilistic graphical modeling framework. Here information from multiple sensors can be fused through joint probability distributions borrowing notions of causality. These notions of causality are used for deriving intelligent fusion rules that can either help in performance monitoring/anomaly detection or data abstraction by hierarchically combining information across multiple sensors. Finally we propose a novel modular approach for fusion of information from vision (RGB images) and language modalities (user input text) in order for a robot to follow and execute natural language instructions in a household environment. Our proposed technique achieves state of the art accuracy in the popular, realistic Ai2Thor environment based on a Unity based simulator.