One Line To Rule Them All: Generating LO-Shot Soft-Label Prototypes

Abstract

Increasingly large datasets are rapidly driving up the computational costs of machine learning. Prototype generation methods aim to create a small set of synthetic observations that accurately represent a training dataset but greatly reduce the computational cost of learning from it. Assigning soft labels to prototypes can allow increasingly small sets of prototypes to accurately represent the original training dataset. Although foundational work on 'less than one'-shot learning has proven the theoretical plausibility of learning with fewer than one observation per class, developing practical algorithms for generating such prototypes remains an unexplored territory. We propose a novel, modular method for generating soft-label prototypical lines that still maintains representational accuracy even when there are fewer prototypes than the number of classes in the data. In addition, we propose the Hierarchical Soft-Label Prototype k-Nearest Neighbor classification algorithm based on these prototypical lines. We show that our method maintains high classification accuracy while greatly reducing the number of prototypes required to represent a dataset, even when working with severely imbalanced and difficult data. Our code is available at https://github.com/ilia10000/SLkNN.

Using Compositionality to Learn Many Categories from Few Examples

“…Previous research in both human and machine learning has treated one-shot learning, where the participant must learn a new concept from a single example, as the limit on sample-efficiency in supervised learning settings (Tiedemann et al, 2022;Fei-Fei et al, 2006a). Recent research in machine learning has shown that it is theoretically possible to learn more novel concepts than the number of presented examples, so-called less-than-one-shot (LO-shot) learning (Sucholutsky & Schonlau, 2021a;Sucholutskv et al, 2021), by associating examples with "soft labels" that describe their closeness to each concept as opposed to traditionally-used "hard labels" which associate each example with a single concept. LO-shot learning has recently been replicated in humans in a study that showed that participants presented with two examples of novel stimuli paired with soft labels relating them to three categories could infer the structure of those three categories (Malaviya et al, 2022).…”

“…There are some limited pieces of empirical evidence that machines (Sucholutsky & Schonlau, 2021b;Sucholutskv et al, 2021) and humans (Malaviya et al, 2022) can perform LO-shot learning by leveraging soft labels that encode the relationship between each example and every known class. While people can somewhat understand and produce soft labels, and can even use them to communicate beliefs to AI systems (Collins, Barker, et al, 2023), they often find these labels unintuitive and difficult to interpret even in simple visual settings .…”