Information Preserving Dimensionality Reduction

Kushagra, Shrinu; Ben-David, Shai

doi:10.1007/978-3-319-24486-0_16

Cited by 1 publication

(2 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We associate with Z a distance function d Z : Z × Z → R, and for convenience write d Z (X 1 , X 2 ) for d Z (g(X 1 ), g(X 2 )). To characterize the behavior of label distributions with respect to the embedding space, we adapt the idea of conditional probabilistic Lipschitzness in [35,47] to present a variant of probabilistic bi-Lipschitzness: Definition 1. (L, M )-Lipschitzness.…”

Section: Preliminariesmentioning

confidence: 99%

“…Theoretically, we provide a sequence of results evaluating how much signal we can extract from pre-trained embeddings using EPOXY. First, we define a notion of probabilistic Lipschitzness [35] to describe the label space's smoothness, with which we can tightly characterize the improvement in rate of convergence and generalization error of EPOXY in the size of the (unlabeled) data over WS without source extensions. We show that improvement depends on two factors: the increase in coverage from extending weak labels, and the accuracy of the extension.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Train and You'll Miss It: Interactive Model Iteration with Weak Supervision and Pre-Trained Embeddings

Chen,

Fu,

Sala

et al. 2020

Preprint

View full text Add to dashboard Cite

Our goal is to enable machine learning systems to be trained interactively. This requires models that perform well and train quickly, without large amounts of hand-labeled data. We take a step forward in this direction by borrowing from weak supervision (WS), wherein models can be trained with noisy sources of signal instead of hand-labeled data. But WS relies on training downstream deep networks to extrapolate to unseen data points, which can take hours or days. Pre-trained embeddings can remove this requirement. We do not use the embeddings as features as in transfer learning (TL), which requires fine-tuning for high performance, but instead use them to define a distance function on the data and extend WS source votes to nearby points. Theoretically, we provide a series of results studying how performance scales with changes in source coverage, source accuracy, and the Lipschitzness of label distributions in the embedding space, and compare this rate to standard WS without extension and TL without fine-tuning. On six benchmark NLP and video tasks, our method outperforms WS without extension by 4.1 points, TL without fine-tuning by 12.8 points, and traditionally-supervised deep networks by 13.1 points, and comes within 0.7 points of state-of-the-art weakly-supervised deep networks-all while training in less than half a second.

show abstract

Section: Preliminariesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%