A Rigorous Theory of Conditional Mean Embeddings

The linear conditional expectation in Hilbert space

“…In machine learning, the method of conditional mean embedding (CME; [16,33]) applies the conditioning formula (1.3) to random variables embedded into RKHSs, where it becomes exact (i.e., γ A U |V = γ U |V ) under certain conditions; see [22]. Section 5 provides an alternative derivation of the CME formula based on linear conditional expectations and thereby a natural justification of CMEs based in BLUEs; to the best of our knowledge, this connection has not been made before.…”

“…In most practical applications the means and (cross-)covariance operators of U and V are not accessible explicitly, but have to be approximated empirically from data (in the simplest case, from independent and identically distributed samples (u n , v n ) ∼ P UV , where P UV denotes the joint distribution of U and V ). Since the Moore-Penrose pseudo-inverse C † V shows an unstable behaviour when approximated empirically [22], Section SM2, it is typically replaced by its regularised version…”

“…which holds under appropriate technical assumptions [22]. Formula (5.1) can be interpreted as saying that application of the Kálmán update or BLUE formulae to RKHS embeddings of random variables realises the embedding of conditional distributions.…”

“…Here, with apologies for the large notational overhead relative to the brevity of the results in Section 5.2, we introduce the precise technical assumptions and notation needed for the validity of the CME approach; see [22], Section 2, for a detailed exposition.…”

“…Our second contribution is a simple alternative derivation and intuitive explanation of the widely used formula for the conditional mean embedding (CME), a method used in machine learning to perform the conditioning of random variables by embedding them into RKHSs, where it reduces to an affine transformation similar to (1.3) [16,22,33]. This result follows almost directly from the fact that, by the reproducing property, E[U |V ] coincides with its best affine approximation E A [U |V ].…”