Compositional Generative Mapping for Tree-Structured Data—Part I: Bottom-Up Probabilistic Modeling of Trees

Abstract

We introduce a novel compositional (recursive) probabilistic model for trees that defines an approximated bottom-up generative process from the leaves to the root of a tree. The proposed model defines contextual state transitions from the joint configuration of the children to the parent nodes. We argue that the bottom-up context postulates different probabilistic assumptions with respect to a top-down approach, leading to different representational capabilities. We discuss classes of applications that are best suited to a bottom-up approach. In particular, the bottom-up context is shown to better correlate and model the co-occurrence of substructures among the child subtrees of internal nodes. A mixed memory approximation is introduced to factorize the joint children-to-parent state transition matrix as a mixture of pairwise transitions. The proposed approach is the first practical bottom-up generative model for tree-structured data that maintains the same computational class of its top-down counterpart. Comparative experimental analyses exploiting synthetic and real-world datasets show that the proposed model can deal with deep structures better than a top-down generative model. The model is also shown to better capture structural information from real-world data comprising trees with a large out-degree. The proposed bottom-up model can be used as a fundamental building block for the development of other new powerful models.

Integrating bi-directional contexts in a generative kernel for trees

“…The bottom-up HTMM (BU) [5], [6] defines a generative process propagating from the leaves to the root of the tree, which allows nodes to collect dependency information from each child subtree. The BU implements a generative process that composes the child subtrees of each node in the tree in a recursive fashion.…”

“…The problem with the formulation in (3) is that it becomes computationally impractical for trees other than binary, since the size of the joint conditional transition distribution is order of C L+1 , where L is the node outdegree. In [6], this has been addressed by introducing a scalable switching parent approximation that factorizes (3) as a mixture of L pairwise child-parent transitions. The resulting BU joint distribution is…”

“…The resulting algorithm is a two-stage iterative procedure, known also as the BaumWelch algorithm [16] in the context of Markov chains, which allows to efficiently compute a solution to the loglikelihood maximization problem. At the E-step, it estimates the posterior of the indicator variables introduced in the completed log-likelihood, while, at the M-Step, it exploits such posteriors to update the model parameters θ. Posterior estimation is the most critical part of the algorithm and can be efficiently computed by message passing upwards and downwards on the structure of the nodes' dependency graph [6], [17]: this procedure is an extension to trees of the Forward-Backward inference algorithm for HMMs on sequences [3].…”

“…In other words, the hidden state space of BU provides a summarized view of the subtrees occurring in the data, where each hidden state identifies a cluster of similar structures, while TD provides a summarized view where each hidden state clusters similar root-to-node paths. In [6] it has been shown that the conditional dependence relationships introduced by the BU context allow, in general, to capture more discriminant details on the tree structures with respect to a TD context. Nevertheless, we are convinced that the TD approach models a different form of structural information with respect to BU and that the integration of the two approaches can yield to more accurate and effective learning models for trees.…”

“…Consider a hidden Markov model (HMM) [3] in the sequential domain: the inversion of the sequence parsing direction changes the orientation of the causal relationships in the model but the two dynamic Bayesian Networks (DBNs) originated by the two different contexts are equivalent from the point of view of the Markov properties [4]. A similar generative model for trees, on the other hand, is more deeply influenced by a change in the context, such that bottom-up [5], [6] and top-down [7], [8] approaches are characterized by different causal relationships that produce two different DBNs from a Markov-equivalence point of view [6]. This creates differences in the local Markov properties of the two models which influence the way the nodes exchange information during inference and learning [4], and ultimately determine what structural information is captured by the hidden states associated to the tree nodes.…”