Impact of Training Dataset Size on Neural Answer Selection Models

Abstract

It is held as a truism that deep neural networks require large datasets to train effective models. However, large datasets, especially with high-quality labels, can be expensive to obtain. This study sets out to investigate (i) how large a dataset must be to train well-performing models, and (ii) what impact can be shown from fractional changes to the dataset size. A practical method to investigate these questions is to train a collection of deep neural answer selection models using fractional subsets of varying sizes of an initial dataset. We observe that dataset size has a conspicuous lack of effect on the training of some of these models, bringing the underlying algorithms into question. This is the authors version of the work. It is posted here for your personal use. The definitive version is published in:

Neural (Knowledge Graph) Question Answering Using Synthetic Training Data

“…In my ECIR'19 short paper [12], some bene ts from increased training data volumes were demonstrated, but increasing data volume did not show a large di erence. Some models quickly reached a plateau of performance within the range of available training data volumes, indicating memorization of the smaller dataset.…”

“…In my ECIR'19 short paper [12], the e ects of varying data size on training neural models for answer sentence selection were studied. In my ICTIR'20 full paper [13], the impact of template-based synthetic data generation on training neural knowledge graph question answering (KGQA) models was investigated.…”

“…In my ECIR'19 short paper [12], 10 neural architectures for answer sentence selection (or, more generally, text matching) were used to train models with di erent fractions of the original training dataset. The models were trained on WikiQA [24] data, using as the training split 10%, 25%, 50%, 75%, or 100% of the original training dataset.…”

Requirements and Limitations of Thermal Drones for Effective Search and Rescue in Marine and Coastal Areas

“…However, computer vision based machine learning algorithms-those which are able to identify objects in images-require a large amount of training data to be able to recognise any specific object. The accuracy of these algorithms deteriorates rapidly if insufficient training data are available [26][27][28], and, in some cases, simpler statistical methods have been shown to perform better than deep learning algorithms for object recognition [29].…”

“…After splitting the data into training, testing and verification sets, this would leave a small number of meaningfully different images to train with (i.e., not temporally adjacent in the footage, thus forming essentially the same image). Previous work has shown that small training dataset sizes lead to dramatic drops in classification accuracy [26][27][28]. With a small number of experiments, a machine learning system would only recognise the specific humans in the specific scenarios we tested with, and it would be difficult to be certain that the reported performance would in fact be meaningful in a wider range of real world scenarios.…”