The relative performance of ensemble methods with deep convolutional neural networks for image classification

Ju, Cheng; Bibaut, Aurélien F.; Laan, Mark J. van der

doi:10.1080/02664763.2018.1441383

Cited by 285 publications

(220 citation statements)

References 47 publications

Supporting

Mentioning

205

Contrasting

Unclassified

Order By: Relevance

“…This design strategy of using independently-optimized diffractive networks is in fact similar to ensemble methods [29], [30] that are frequently used in machine learning literature. Figure 6 summarizes the blind testing accuracies achieved by this strategy using either non-differential or differential diffractive neural networks, i.e., D( [10,0], [1,5,40k]) or D( [10,10], [1,5,40k]).…”

Section: Resultsmentioning

confidence: 98%

Class-specific differential detection in diffractive optical neural networks improves inference accuracy

et al. 2019

View full text Add to dashboard Cite

Optical computing provides unique opportunities in terms of parallelization, scalability, power efficiency and computational speed, and has attracted major interest for machine learning. Diffractive deep neural networks have been introduced earlier as an optical machine learning framework that uses task-specific diffractive surfaces designed by deep learning to all-optically perform inference, achieving promising performance for object classification and imaging. Here we demonstrate systematic improvements in diffractive optical neural networks based on a differential measurement technique that mitigates the strict non-negativity constraint of light intensity. In this differential detection scheme, each class is assigned to a separate pair of detectors, behind a diffractive optical network, and the class inference is made by maximizing the normalized signal difference between the photodetector pairs. Using this differential detection scheme, involving 10 photodetector pairs behind 5 diffractive layers with a total of 0.2 Million neurons, we numerically achieved blind testing accuracies of 98.54%, 90.54% and 48.51% for MNIST, Fashion-MNIST and grayscale CIFAR-10 datasets, respectively. Moreover, by utilizing the inherent parallelization capability of optical systems, we reduced the cross-talk and optical signal coupling between the positive and negative detectors of each class by dividing the optical path into two jointly-trained diffractive neural networks that work in parallel. We further made use of this parallelization approach, and divided individual classes in a target dataset among multiple jointly-trained diffractive neural networks. Using this class-specific differential detection in jointly-optimized diffractive neural networks that operate in parallel, our simulations achieved blind testing accuracies of 98.52%, 91.48% and 50.82% for MNIST, Fashion-MNIST and grayscale CIFAR-10 datasets, respectively, coming close to the performance of some of the earlier generations of all-electronic deep neural networks, e.g., LeNet, which achieves classification accuracies of 98.77%, 90.27%, and 55.21% corresponding to the same datasets, respectively. In addition to these jointly-optimized diffractive neural networks, we also independently-optimized multiple diffractive networks and utilized them in a way that is similar to ensemble methods practiced in machine learning; using 3 independently-optimized differential diffractive neural networks that optically project their light onto a common output/detector plane, we numerically achieved blind testing accuracies of 98.59%, 91.06% and 51.44% for MNIST, Fashion-MNIST and grayscale CIFAR-10 datasets, respectively. Through these systematic advances in designing diffractive neural networks, the reported classification accuracies set the state-of-the-art for an all-optical neural network design, and the presented framework might be useful to bring optical neural network-based low power solutions for various machine learning applications and help us design new computational c...

show abstract

Section: Resultsmentioning

confidence: 98%

Class-specific differential detection in diffractive optical neural networks improves inference accuracy

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Well-known ensemble techniques include boosting, bagging and stacking. Stacking combines the outputs of a set of base learners and lets another algorithm, referred to as the meta-learner, make the final predictions [14]. A super learner is another method that calculates the final predictions by finding the optimal weights of the base learners by minimizing a loss function based on the cross-validated output of the learners [14].…”

Section: Ensemble Methodsmentioning

confidence: 99%

“…The most common ensemble method used for neural networks is average voting that generates posterior labels by calculating the average of the softmax class probabilities or predicted labels for all the base learners [14]. Majority voting is another ensemble approach that counts the predicted labels from all the base learners and reports the label with the maximum number of votes as the final prediction.…”

Section: Ensemble Methodsmentioning

confidence: 99%

Ensemble Convolutional Neural Networks for Mode Inference in Smartphone Travel Survey

Yazdizadeh

Patterson

Farooq

2020

IEEE Trans. Intell. Transport. Syst.

View full text Add to dashboard Cite

We develop ensemble Convolutional Neural Networks (CNNs) to classify the transportation mode of trip data collected as part of a large-scale smartphone travel survey in Montreal, Canada. Our proposed ensemble library is composed of a series of CNN models with different hyper-parameter values and CNN architectures. In our final model, we combine the output of CNN models using "average voting", "majority voting" and "optimal weights" methods. Furthermore, we exploit the ensemble library by deploying a Random Forest model as a meta-learner. The ensemble method with random forest as meta-learner shows an accuracy of 91.8% which surpasses the other three ensemble combination methods, as well as other comparable models reported in the literature. The "majority voting" and "optimal weights" combination methods result in prediction accuracy rates around 89%, while "average voting" is able to achieve an accuracy of only 85%.

show abstract

“…Other ensemble methods like Super Learner (Ju et al, 2017) should be tried as well. Since label sparsity is at the heart of the performance difference between ChatScript and the CNN models, a more direct way to deal with lack of training examples (possibly obviating the need for a hand-crafted system like ChatScript) could be to automatically generate paraphrases to augment available data, potentially with a content author in the loop; we are currently exploring strategies for doing so.…”

Section: Discussion and Future Workmentioning

confidence: 99%

“…Previous research has shown that ensembling models improves performance (Sakaguchi et al, 2015b;Ju et al, 2017;He et al, 2017). We train different models with different splits of training and develop data, and ensemble them together.…”

Section: Ensemble Methodsmentioning

confidence: 99%

Combining CNNs and Pattern Matching for Question Interpretation in a Virtual Patient Dialogue System

Jin

White

Jaffe

et al. 2017

Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

For medical students, virtual patient dialogue systems can provide useful training opportunities without the cost of employing actors to portray standardized patients. This work utilizes word-and character-based convolutional neural networks (CNNs) for question identification in a virtual patient dialogue system, outperforming a strong word-and characterbased logistic regression baseline. While the CNNs perform well given sufficient training data, the best system performance is ultimately achieved by combining CNNs with a hand-crafted pattern matching system that is robust to label sparsity, providing a 10% boost in system accuracy and an error reduction of 47% as compared to the pattern-matching system alone.

show abstract

The relative performance of ensemble methods with deep convolutional neural networks for image classification

Cited by 285 publications

References 47 publications

Class-specific differential detection in diffractive optical neural networks improves inference accuracy

Class-specific differential detection in diffractive optical neural networks improves inference accuracy

Ensemble Convolutional Neural Networks for Mode Inference in Smartphone Travel Survey

Combining CNNs and Pattern Matching for Question Interpretation in a Virtual Patient Dialogue System

Contact Info

Product

Resources

About