Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures

Abstract

Model-based methods and deep neural networks have both been tremendously successful paradigms in machine learning. In model-based methods, problem domain knowledge can be built into the constraints of the model, typically at the expense of difficulties during inference. In contrast, deterministic deep neural networks are constructed in such a way that inference is straightforward, but their architectures are generic and it is unclear how to incorporate knowledge. This work aims to obtain the advantages of both approaches. To do so, we start with a modelbased approach and an associated inference algorithm, and unfold the inference iterations as layers in a deep network. Rather than optimizing the original model, we untie the model parameters across layers, in order to create a more powerful network. The resulting architecture can be trained discriminatively to perform accurate inference within a fixed network size. We show how this framework allows us to interpret conventional networks as mean-field inference in Markov random fields, and to obtain new architectures by instead using belief propagation as the inference algorithm. We then show its application to a non-negative matrix factorization model that incorporates the problem-domain knowledge that sound sources are additive. Deep unfolding of this model yields a new kind of non-negative deep neural network, that can be trained using a multiplicative backpropagation-style update algorithm. We present speech enhancement experiments showing that our approach is competitive with conventional neural networks despite using far fewer parameters.

FAID Diversity via Neural Networks

“…D 1 can be designed by various methods, such as the Density Evolution [15] and quantized neural networks [6], [7]. We then use the sub-graph expansion-contraction [12] (as will be introduced in Section III-B) to determine the most problematic error patterns In the Initialization, the sub-graph expansion-contraction is applied for D 1 to find the set of the most problematic error patterns E (1) , and E (1) is set to be the first training set. In the i-th round of Sequential design, the RQNN is trained with the training set E (i) to design the FAID D i+1 .…”

“…that cannot be corrected by D 1 . Denote the set consisting of most problematic error patterns as E (1) and the guaranteed correction capability of D 1 as t, then for any error pattern e ∈ E (1) , w(e) = t + 1. In particular, E (1) will be used as the initial training set in the next step.…”

“…In the sequential design, we use a recurrent quantized neural network to construct the rest of the FAIDs with the objective of correcting as many error patterns in E (1) as possible. We train the RQNN in multiple rounds, with one FAID per round.…”

“…This process continues until E (i) becomes an empty set or a predefined maximum number of rounds N D − 1 has been reached. If E (i) eventually becomes empty, the designed FAID diversity can correct all error patterns with weight up to t + 1 in E (1) . Otherwise, the process have completed N D − 1 rounds and the last training set E (N D ) is a nonempty set.…”

“…One popular idea, known as model-driven neural networks (NNs), is to combine the model knowledge (or the prototype algorithms) and the NN in conjunction with optimization techniques of NNs to improve the model. In the context of iterative decoding of error correction codes, the deep unfolding framework [1] is particularly attractive as it naturally unfolds the decoding iterations over the Tanner graph into a deep neural network. The activation functions are defined by the prototype decoding rules that mimic the message-passing process [2].…”