Multi-loss Regularized Deep Neural Network

Xu, Chunyan; Lu, Canyi; Liang, Xiaodan; Gao, Junbin; Zheng, Wei; Wang, Tianjiang; Yan, Shuicheng

doi:10.1109/tcsvt.2015.2477937

Cited by 79 publications

(46 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additional works where state-of-the-art results were obtained without data augmentation include those by McFonnell and Vladusich [35], who reported a test error rate of 0.37% using a fast-learning shallow convolutional neural network, Mairal et al [36], who achieved 0.39% using convolutional kernel networks, Xu et al [37], who explored multi-loss regularization in CNNs obtaining an error rate of 0.42%, and Srivastava et al [38] used so-called convolutional "highway" networks (inspired by LSTM recurrent networks) to achieve an error rate of 0.45%.…”

Section: State Of the Artmentioning

confidence: 99%

A Survey of Handwritten Character Recognition with MNIST and EMNIST

2019

View full text Add to dashboard Cite

This paper summarizes the top state-of-the-art contributions reported on the MNIST dataset for handwritten digit recognition. This dataset has been extensively used to validate novel techniques in computer vision, and in recent years, many authors have explored the performance of convolutional neural networks (CNNs) and other deep learning techniques over this dataset. To the best of our knowledge, this paper is the first exhaustive and updated review of this dataset; there are some online rankings, but they are outdated, and most published papers survey only closely related works, omitting most of the literature. This paper makes a distinction between those works using some kind of data augmentation and works using the original dataset out-of-the-box. Also, works using CNNs are reported separately; as they are becoming the state-of-the-art approach for solving this problem. Nowadays, a significant amount of works have attained a test error rate smaller than 1% on this dataset; which is becoming non-challenging. By mid-2017, a new dataset was introduced: EMNIST, which involves both digits and letters, with a larger amount of data acquired from a database different than MNIST's. In this paper, EMNIST is explained and some results are surveyed.

show abstract

Section: State Of the Artmentioning

confidence: 99%

A Survey of Handwritten Character Recognition with MNIST and EMNIST

2019

View full text Add to dashboard Cite

show abstract

“… Multi-loss function Different loss functions lead the networks to reach different local minima. The analysis of different losses also showed they have their own strengths and limitations (Janocha & Czarnecki, 2017;Rosasco, De Vito, Caponnetto, Piana, & Verri, 2004;C. Xu et al, 2016), while the question arises as how to combine these advantages in a unified system without the heavy computational load required for independent runs of different loss functions and combining them.…”

Section:  Ce Loss and Csd Lossmentioning

confidence: 99%

“…From another point of view, different loss functions have complementary advantages and limitations (Janocha & Czarnecki, 2017;C. Xu et al, 2016).…”

Section: Introductionmentioning

confidence: 99%

Multi-representational learning for Offline Signature Verification using Multi-Loss Snapshot Ensemble of CNNs

Masoudnia

Mersa

Araabi

et al. 2019

Expert Systems with Applications

View full text Add to dashboard Cite

Offline Signature Verification (OSV) is a challenging pattern recognition task, especially in the presence of skilled forgeries that are not available during training. This study aims to tackle its challenges and meet the substantial need for generalization for OSV by examining different loss functions for Convolutional Neural Network (CNN). We adopt our new approach to OSV by asking two questions: 1. which classification loss provides more generalization for feature learning in OSV? , and 2. How integration of different losses into a unified multi-loss function lead to an improved learning framework?These questions are studied based on analysis of three loss functions, including cross entropy, Cauchy-Schwarz divergence, and hinge loss. According to complementary features of these losses, we combine them into a dynamic multi-loss function and propose a novel ensemble framework for simultaneous use of them in CNN. Our proposed Multi-Loss Snapshot Ensemble (MLSE) consists of several sequential trials. In each trial, a dominant loss function is selected from the multi-loss set, and the remaining losses act as a regularizer. Different trials learn diverse representations for each input based on signature identification task. This multi-representation set is then employed for the verification task. An ensemble of SVMs is trained on these representations, and their decisions are finally combined according to the selection of most generalizable SVM for each user.We conducted two sets of experiments based on two different protocols of OSV, i.e., writer-dependent and writer-independent on three signature datasets: GPDS-Synthetic, MCYT, and UT-SIG. Based on the writerdependent OSV protocol, On UT-SIG, we achieved 6.17% Equal Error Rate (EER) which showed substantial improvement over the best EER in the literature, 9.61%. Our method surpassed state-of-the-arts by 2.5% on GPDS-Synthetic, achieving 6.13%. Our result on MCYT was also comparable to the best previous results.The second set of experiments examined the robustness of our proposed method to the arrival of new users enrolled in the OSV system based on the writer-independent protocol. The results also confirmed that our proposed system efficiently performed the verification of new users enrolled in the OSV system.

show abstract

“…Ren et al [25] proposed a Region Proposal Network (RPN) that shared full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. In [26], a Multi-Loss regularized Deep Neural Network (ML-DNN) framework was proposed, which exploited multiple loss functions with different theoretical motivations to mitigate overfitting during semantic concept learning. He et al [27] proposed a residual learning framework to alleviate the training of neural networks.…”

Section: Related Workmentioning

confidence: 99%

Combining Convolutional Neural Network and Markov Random Field for Semantic Image Retrieval

Huang

Huang³

et al. 2018

Advances in Multimedia

Self Cite

View full text Add to dashboard Cite

With the rapidly growing number of images over the Internet, efficient scalable semantic image retrieval becomes increasingly important. This paper presents a novel approach for semantic image retrieval by combining Convolutional Neural Network (CNN) and Markov Random Field (MRF). As a key step, image concept detection, that is, automatically recognizing multiple semantic concepts in an unlabeled image, plays an important role in semantic image retrieval. Unlike previous work that uses single-concept classifiers one by one, we detect semantic multiconcept by using a multiconcept scene classifier. In other words, our approach takes multiple concepts as a holistic scene for multiconcept scene learning. Specifically, we first train a CNN as a concept classifier, which further includes two types of classifiers: a single-concept fully connected classifier that is best suited to single-concept detection and a multiconcept scene fully connected classifier that is good for holistic scene detection. Then we propose an MRF-based late fusion approach that is able to effectively learn the semantic correlation between the single-concept classifier and multiconcept scene classifier. Finally, the semantic correlation among the subconcepts of images is cought to further improve detection precision. In order to investigate the feasibility and effectiveness of our proposed approach, we conduct comprehensive experiments on two publicly available image databases. The results show that our proposed approach outperforms several state-of-the-art approaches.

show abstract

Multi-loss Regularized Deep Neural Network

Cited by 79 publications

References 21 publications

A Survey of Handwritten Character Recognition with MNIST and EMNIST

A Survey of Handwritten Character Recognition with MNIST and EMNIST

Multi-representational learning for Offline Signature Verification using Multi-Loss Snapshot Ensemble of CNNs

Combining Convolutional Neural Network and Markov Random Field for Semantic Image Retrieval

Contact Info

Product

Resources

About