Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

Abstract

The success of deep learning in vision can be attributed to: (a) models with high capacity; (b) increased computational power; and (c) availability of large-scale labeled data. Since 2012, there have been significant advances in representation capabilities of the models and computational capabilities of GPUs. But the size of the biggest dataset has surprisingly remained constant. What will happen if we increase the dataset size by 10× or 100×? This paper takes a step towards clearing the clouds of mystery surrounding the relationship between 'enormous data' and visual deep learning. By exploiting the JFT-300M dataset which has more than 375M noisy labels for 300M images, we investigate how the performance of current vision tasks would change if this data was used for representation learning. Our paper delivers some surprising (and some expected) findings. First, we find that the performance on vision tasks increases logarithmically based on volume of training data size. Second, we show that representation learning (or pretraining) still holds a lot of promise. One can improve performance on many vision tasks by just training a better base model. Finally, as expected, we present new state-of-theart results for different vision tasks including image classification, object detection, semantic segmentation and human pose estimation. Our sincere hope is that this inspires vision community to not undervalue the data and develop collective efforts in building larger datasets.

Rescuing Collective Wisdom when the Average Group Opinion Is Wrong

“…Machine learning tools and improved ability to gather data provides the opportunity to learn more sophisticated WOC methods in a datadriven fashion (Rokach, 2010;Bachrach et al, 2012;Polikar, 2012;Sun et al, 2017). We are likely to learn much more about effective strategies of opinion aggregation through their widespread adoption.…”

“…Recent technological advances have also opened up the possibility of gathering very large datasets from which collective wisdom can be extracted (Sun et al, 2017). Large datasets allow researchers to consider and reliably test increasingly complex methodologies of opinion aggregation.…”

A deep learning approach to estimate chemically-treated collagenous tissue nonlinear anisotropic stress-strain responses from microscopy images

“…tissue image in this application) are very unique compared to those of the training data, then the trained machine learning models will not recognize the input. As suggested by a recent work on the relationship between deep learning and training data [50], this problem can be solved by using more training data and fine tuning the neural network in our future work. In case of having extremely unique samples, a rejection option [51, 52] can be incorporated to the machine learning models, and the enhanced models can refuse to make a decision if the patterns of input are very different from those of the training data.…”

The Enemy of Good: Estimating the Cost of Waiting for Nearly Perfect Automated Vehicles

“…(Pratt, 2017) Simultaneously, research on technology improvement across industries suggests that there is a power-law relationship between production and performance: A doubling of production leads to a constant improvement in performance (as measured by cost or other characteristics) (Nagy et al, 2013). A similar relationship may exist for the performance of machine learning algorithms and data sets (Sun et al, 2017). For HAVs, this suggests that achieving gains that some might consider "near perfect" may take much more effort and time than reaching better-than-average human performance, which may itself be still out of reach.…”