Measuring Multi-Source Redundancy in Factor Graphs

Abstract

In this paper, we define a new measure of the redundancy of information from a fault tolerance perspective. The partial information decomposition (PID) emerged last decade as a framework for decomposing the multi-source mutual information I(T ; X1, ..., Xn) into atoms of redundant, synergistic, and unique information. It built upon the notion of redundancy/synergy from McGill's interaction information [1]. Separately, the redundancy of system components has served as a principle of fault tolerant engineering, for sensing, routing, and control applications. Here, redundancy is understood as the level of duplication necessary for the fault tolerant performance of a system. With these two perspectives in mind, we propose a new PID-based measure of redundancy Ift, based upon the presupposition that redundant information is robust to individual source failures. We demonstrate that this new measure satisfies the common PID axioms from [2]. In order to do so, we establish an orderreversing correspondence between collections of source-fallible instantiations of a system, on the one hand, and the PID lattice from [2], on the other.

Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks

“…Recent work developed a differentiable PID measure [ 17 , 18 ], which has been used to specify local learning rules optimizing for interpretable information-processing goals in neural networks [ 19 ]. Other studies have used PID for feature selection [ 20 ], analyzing training dynamics in convolutional neural networks [ 21 ], estimating redundancy in factor graphs [ 22 ], and studying representational complexity in ANNs [ 23 ], among other work [ 24 , 25 ].…”