Learning What Data to Learn

Yang, Fan; Tian, Fei; Qin, Tao; Bian, Jiang; Liu, Tie-Yan

doi:10.48550/arxiv.1702.08635

Cited by 24 publications

(20 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on the influence function, Wang et al [28] locate samples which are not helpful after the first round of training, and retrain the model from scratch without these samples. For the same purpose, Ren et al [21] utilize meta learning method to assign weights based on the gradient direction of samples, and Fan et al [5] explores deep reinforcement learning to automatically select samples in the training process. Sample re-weighting strategies are also applied in the object detection task [22,14,13].…”

Section: Sample Re-weighting In Deep Learningmentioning

confidence: 99%

Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association

Wen

Xu²,

Jiang

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Nowadays, we have witnessed the early progress on learning the association between voice and face automatically, which brings a new wave of studies to the computer vision community. However, most of the prior arts along this line (a) merely adopt local information to perform modality alignment and (b) ignore the diversity of learning difficulty across different subjects. In this paper, we propose a novel framework to jointly address the above-mentioned issues. Targeting at (a), we propose a two-level modality alignment loss where both global and local information are considered. Compared with the existing methods, we introduce a global loss into the modality alignment process. The global component of the loss is driven by the identity classification. Theoretically, we show that minimizing the loss could maximize the distance between embeddings across different identities while minimizing the distance between embeddings belonging to the same identity, in a global sense (instead of a mini-batch). Targeting at (b), we propose a dynamic reweighting scheme to better explore the hard but valuable identities while filtering out the unlearnable identities. Experiments show that the proposed method outperforms the previous methods in multiple settings, including voice-face matching, verification and retrieval.

show abstract

Section: Sample Re-weighting In Deep Learningmentioning

confidence: 99%

Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association

Wen

Xu²,

Jiang

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…Data efficiency has been studied for the traditional training in literature [20,21,22,23]. It has been proven that the amount of information provided by each training example to a network is different, and the difficulty of learning examples also varies.…”

Section: Exploring Data Efficiency In Sparse Trainingmentioning

confidence: 99%

“…According to prior works [20,21,22,23], the unforgettable examples are generally considered as less informative and easy to be learned. The…”

Section: F Data-efficient Training F1 Basic Conceptsmentioning

confidence: 99%

“…On top of that, the paper proposes to employ data efficiency for further acceleration of sparse training. The prior works [20,21,22,23] show that the amount of information provided by each training example is different, and the hardness of having an example correctly learned is also different. Some training examples can be learned correctly early in the training stage and will never be "forgotten" (i.e., misclassified) again.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge

Yuan¹,

Ma²,

Niu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, a new trend of exploring sparsity for accelerating neural network training has emerged, embracing the paradigm of training on the edge. This paper proposes a novel Memory-Economic Sparse Training (MEST) framework targeting for accurate and fast execution on edge devices. The proposed MEST framework consists of enhancements by Elastic Mutation (EM) and Soft Memory Bound (&S) that ensure superior accuracy at high sparsity ratios. Different from the existing works for sparse training, this current work reveals the importance of sparsity schemes on the performance of sparse training in terms of accuracy as well as training speed on real edge devices. On top of that, the paper proposes to employ data efficiency for further acceleration of sparse training. Our results suggest that unforgettable examples can be identified in-situ even during the dynamic exploration of sparsity masks in the sparse training process, and therefore can be removed for further training speedup on edge devices. Comparing with state-of-the-art (SOTA) works on accuracy, our MEST increases Top-1 accuracy significantly on ImageNet when using the same unstructured sparsity scheme. Systematical evaluation on accuracy, training speed, and memory footprint are conducted, where the proposed MEST framework consistently outperforms representative SOTA works. Our codes are publicly available at: https://github.com/boone891214/MEST.Recently, a new trend of exploring sparsity for training acceleration of neural networks has emerged to embrace the promising training-on-the-edge paradigm. The first works in this direction use the pruning-at-initialization approach such as SNIP [9] and GraSP [10] that first obtains a fixed sparse model structure and then follows with a traditional training process. However, the whole process is still computation-and memory-intensive, and therefore not compatible with the end-to-end edge training paradigm. Such a sparse training methodology with the pre-fixed structure also faces the problem of compromised accuracy.

show abstract

“…Therefore, many previous neural network approaches exploit successful trajectories by traditional motion planners to create training datasets [19]- [21], [23], [24]. However, all data samples are not equally important for the training of neural networks [26], [27]. Adaptive sampling or boosting can accelerate training and improve performance.…”

Section: Related Workmentioning

confidence: 99%

Learning Continuous Cost-to-Go Functions for Non-holonomic Systems

Huh¹,

Lee²,

Isler³

2021

Preprint

View full text Add to dashboard Cite

This paper presents a supervised learning method to generate continuous cost-to-go functions of non-holonomic systems directly from the workspace description. Supervision from informative examples reduces training time and improves network performance. The manifold representing the optimal trajectories of a non-holonomic system has high-curvature regions which can not be efficiently captured with uniform sampling. To address this challenge, we present an adaptive sampling method which makes use of sampling based planners along with local, closed-form solutions to generate training samples. The cost-to-go function over a specific workspace is represented as a neural network whose weights are generated by a second, higher order network. The networks are trained in an end-to-end fashion. In our previous work, this architecture was shown to successfully learn to generate the cost-to-go functions of holonomic systems using uniform sampling. In this work, we show that uniform sampling fails for non-holonomic systems. However, with the proposed adaptive sampling methodology, our network can generate near-optimal trajectories for nonholonomic systems while avoiding obstacles. Experiments show that our method is two orders of magnitude faster compared to traditional approaches in cluttered environments.

show abstract

Learning What Data to Learn

Cited by 24 publications

References 19 publications

Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association

Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association

MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge

Learning Continuous Cost-to-Go Functions for Non-holonomic Systems

Contact Info

Product

Resources

About