How Much More Data Do I Need? Estimating Requirements for Downstream Tasks

Mahmood, Rafid; Lucas, James; Acuna, David; Li, Daiqing; Philion, Jonah; Álvarez, Jose M.; Yu, Zhiding; Fidler, Sania; Law, Marc T.

doi:10.1109/cvpr52688.2022.00037

Cited by 9 publications

(28 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multi-variate scaling laws have also been considered for some specific tasks, for example in transfer learning from synthetic to real data sets [11]. Finally, Mahmood et al [2] explore data collection by estimating the minimum amount of data needed to meet a given target performance over multiple rounds. Our paper extends these prior studies by developing an optimization problem to minimize the expected total cost of data collected.…”

Section: Related Workmentioning

confidence: 99%

“…For instance, Rosenfeld et al [6] fit power law functions on the performance statistics of small data sets to extrapolate the learning curve with more data. In contrast, Mahmood et al [2] consider estimating data requirements and show that even small errors in a power law model of the learning curve can translate to massively over-or underestimating how much data is needed. Beyond this, different data sources have different costs and scale differently with performance [11][12][13][14].…”

Section: Introductionmentioning

confidence: 99%

“…For instance, annotating segmentation masks for a driving data set takes between 15 to 40 seconds per object. For 100,000 images the annotation could require between 170 and 460 days-equivalent of time [1,2]. On the other hand, collecting too little data may incur future costs and workflow delays from having to collect more later.…”

Section: Introductionmentioning

confidence: 99%

“…Furthermore, our optimization problem immediately generalizes to decisions over multiple data sources (e.g., unlabeled, long-tail, cross-domain, synthetic) that have different costs and impacts on performance. Finally, we demonstrate the value of optimization over naïvely estimating data set requirements (e.g., [2]) for several machine learning tasks and data sets.…”

Section: Introductionmentioning

confidence: 99%

“…(1) We propose the optimal data collection problem in machine learning, which formalizes data collection workflows. (2) We introduce Learn-Optimize-Collect (LOC), a learning-and-optimizing framework that minimizes future collection costs, can be solved via gradient descent, and has analytic solutions in some settings. (3) We generalize the data collection problem and LOC to a multi-variate setting where different types of data have different costs.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Optimizing Data Collection for Machine Learning

Mahmood¹,

Lucas²,

Álvarez³

et al. 2022

Preprint

View full text Add to dashboard Cite

Modern deep learning systems require huge data sets to achieve impressive performance, but there is little guidance on how much or what kind of data to collect. Over-collecting data incurs unnecessary present costs, while under-collecting may incur future costs and delay workflows. We propose a new paradigm for modeling the data collection workflow as a formal optimal data collection problem that allows designers to specify performance targets, collection costs, a time horizon, and penalties for failing to meet the targets. Additionally, this formulation generalizes to tasks requiring multiple data sources, such as labeled and unlabeled data used in semi-supervised learning. To solve our problem, we develop Learn-Optimize-Collect (LOC), which minimizes expected future collection costs. Finally, we numerically compare our framework to the conventional baseline of estimating data requirements by extrapolating from neural scaling laws. We significantly reduce the risks of failing to meet desired performance targets on several classification, segmentation, and detection tasks, while maintaining low total collection costs.

show abstract

Section: Related Workmentioning

confidence: 99%