Disentangled representations via synergy minimization

Abstract

Scientists often seek simplified representations of complex systems to facilitate prediction and understanding. If the factors comprising a representation allow us to make accurate predictions about our system, but obscuring any subset of the factors destroys our ability to make predictions, we say that the representation exhibits informational synergy. We argue that synergy is an undesirable feature in learned representations and that explicitly minimizing synergy can help disentangle the true factors of variation in underlying data. We explore different ways of quantifying synergy, deriving new closed-form expressions in some cases, and then show how to modify learning to produce representations that are minimally synergistic. We introduce a benchmark task to disentangle separate characters from images of words. We demonstrate that Minimally Synergistic (MinSyn) representations correctly disentangle characters while methods relying on statistical independence fail.

MAXENT3D_PID: An Estimator for the Maximum-Entropy Trivariate Partial Information Decomposition

“…Furthermore, the PID measures can also be used as a tool for data analysis and to characterize computational models. This comprises dimensionality reduction via synergy or redundancy minimization [ 19 , 22 ], the study of generative networks that emerge from information maximization constraints [ 78 , 79 ], or explaining the representations in deep networks [ 50 ].…”

“…This information-theoretic approach to study interactions has found many applications to complex systems such as gene networks (e.g., [ 8 , 9 , 10 ]), interactive agents (e.g., [ 11 , 12 , 13 , 14 ]), or neural processing (e.g., [ 15 , 16 , 17 ]). More generally, the nature of the information contained in the inputs determines the complexity of extracting it [ 18 , 19 ], how robust it is to disrupt the system [ 20 ], or how input dimensionality can be reduced without information loss [ 21 , 22 ].…”

Computing the Unique Information

“…We hope that our algorithm will contribute means to test the mutual information decomposition on larger systems than was possible so far, particularly in recent applications of the decomposition, e.g., in neuroscience (Pica et al 2017), representation learning (Steeg et al 2017, Tax et al 2017, robotics (Ghazi-Zahedi and Rauh 2015, Ghazi-Zahedi et al 2017), etc., which so far has been pursued either with only simpler types of measures or for very low-dimensional systems.…”

“…This was followed up by the axiomatic approach from , quantifying the unique, shared, and synergistic information based on ideas from decision theory. In the past couple of years, the latter approach has steadily gained currency with applications ranging, for instance, from quantifying the neural code (Pica et al 2017), to learning deep representations (Steeg et al 2017). We focus on their definitions explained in Section 2.…”

Unique Informations and Deficiencies

“…If the interactions induce redundancy and synergy in equal measure, then the coinformation cannot detect it. Correlational importance, a nonnegative measure introduced in [7] (see also [9]) to quantify the importance of correlations in neural coding is similar in spirit to the synergistic information. However, examples are known [10] when it can exceed the total mutual information.…”