Metric-Based Approaches for Semi-Supervised Regression and Classification

Learning from Labeled and Unlabeled Data

“…It is interesting that, empirically for regression estimation, a quite aggressive multiplicative penalty outperforms an additive penalty based on the same idea, since most regularization strategies currently in use (including Bayesian MAP estimation) employ additive penalties. We further note that the additive criterion in [79], as applied to regression estimation with squared-error loss, has already been suggested in [16]. However, Cataltepe et al [16] do not give a very convincing theoretical motivation.…”

“…However, the technique might still exhibit overfitting, simply because the triangle inequality ( 15) is usually far from tight. An extension of this strategy, called ADJ, attempts a first-order bias correction between the pseudometric d and its estimated version, say d. In a later paper [79], Schuurmans and Southey get rid of the a-priori hierarchy and focus on criteria which are additive or multiplicative combinations of the empirical loss d(h, P (t|x)) and a penalty. They then propose penalties based on the idea that overfitting of h can sometimes be detected by comparing, for some fixed origin function φ, the distances d(h, φ) (which can be estimated reliably using 29 It is not our plan to go into discussions about foundations of Occam's razor here.…”

“…To conclude, adaptive regularization criteria are based on the notion of inputdependent regularization (see subsection 1.3.3 and [83]). While the criteria suggested in [78], [79] are reported to work well on regression estimation tasks, the reported results for classification are less promising. In the latter, the information each labels contains about the latent function is less directly accessible than in regression estimation.…”

“…In the latter, the information each labels contains about the latent function is less directly accessible than in regression estimation. It is also not clear how to inject available prior knowledge into the procedures suggested in [79], but we are very interested in following up this very recent line of research.…”