Hyperparameter Tuning in Bandit-Based Adaptive Operator Selection

Pacula, Maciej; Ansel, Jason; Amarasinghe, Saman; O’Reilly, Una-May

doi:10.1007/978-3-642-29178-4_8

Cited by 7 publications

(4 citation statements)

References 8 publications

(9 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A similar technique was used in [25] in the different context of online operator selection. It is based on an optimal solution to the multi-armed bandit problem [12].…”

Section: Auc Bandit Meta Techniquementioning

confidence: 99%

OpenTuner

Ansel

Kamil

Veeramachaneni

et al. 2014

Proceedings of the 23rd International Conference on Parallel Architectures and Compilation

Self Cite

377

View full text Add to dashboard Cite

Program autotuning has been shown to achieve better or more portable performance in a number of domains. However, autotuners themselves are rarely portable between projects, for a number of reasons: using a domain-informed search space representation is critical to achieving good results; search spaces can be intractably large and require advanced machine learning techniques; and the landscape of search spaces can vary greatly between different problems, sometimes requiring domain specific search techniques to explore efficiently. This paper introduces OpenTuner, a new open source framework for building domain-specific multi-objective program autotuners. OpenTuner supports fully-customizable configuration representations, an extensible technique representation to allow for domain-specific techniques, and an easy to use interface for communicating with the program to be autotuned. A key capability inside OpenTuner is the use of ensembles of disparate search techniques simultaneously; techniques that perform well will dynamically be allocated a larger proportion of tests. We demonstrate the efficacy and generality of OpenTuner by building autotuners for 7 distinct projects and 16 total benchmarks, showing speedups over prior techniques of these projects of up to 2.8× with little programmer effort.

show abstract

“…A similar technique was used in [25] in the different context of online operator selection. It is based on an optimal solution to the multi-armed bandit problem [12].…”

Section: Auc Bandit Meta Techniquementioning

confidence: 99%

OpenTuner

Ansel

Kamil

Veeramachaneni

et al. 2014

Proceedings of the 23rd International Conference on Parallel Architectures and Compilation

Self Cite

377

View full text Add to dashboard Cite

show abstract

“…A separate database which includes performance data for all versions is also maintained for use in autotuning. This will be useful when additional search techniques are implemented such as Bayesian hyperparameter search [78], multi-armed bandit optimization [64] or black-box optimization [34].…”

Section: Compilation Cachementioning

confidence: 99%

Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions

Vasilache,

Zinenko,

Theodoridis

et al. 2018

Preprint

108

View full text Add to dashboard Cite

Deep learning models with convolutional and recurrent networks are now ubiquitous and analyze massive amounts of audio, image, video, text and graph data, with applications in automatic translation, speech-to-text, scene understanding, ranking user preferences, ad placement, etc. Competing frameworks for building these networks such as TensorFlow, Chainer, CNTK, Torch/PyTorch, Caffe1/2, MXNet and Theano, explore different tradeoffs between usability and expressiveness, research or production orientation and supported hardware. They operate on a DAG of computational operators, wrapping high-performance libraries such as CUDNN (for NVIDIA GPUs) or NNPACK (for various CPUs), and automate memory allocation, synchronization, distribution. Custom operators are needed where the computation does not fit existing high-performance library calls, usually at a high engineering cost. This is frequently required when new operators are invented by researchers: such operators suffer a severe performance penalty, which limits the pace of innovation. Furthermore, even if there is an existing runtime call these frameworks can use, it often does not offer optimal performance for a user's particular network architecture and dataset, missing optimizations between operators as well as optimizations that can be done knowing the size and shape of data. Our contributions include (1) a language close to the mathematics of deep learning called Tensor Comprehensions, (2) a polyhedral Just-In-Time compiler to convert a mathematical description of a deep learning DAG into a CUDA kernel with delegated memory management and synchronization, also providing optimizations such as operator fusion and specialization for specific sizes, (3) a compilation cache populated by an autotuner. In particular, we demonstrate the suitability of the polyhedral framework to construct a domain-specific optimizer effective on state-of-the-art deep learning models on GPUs. Our flow reaches up to 4× speedup over NVIDIA libraries on kernels relevant to the Machine Learning Community, and on an actual model used in production at Facebook. It is integrated with mainstream frameworks Caffe2 (production-oriented), PyTorch (research-oriented), through the ATen asynchronous tensor library.

show abstract

“…The Petabricks project [104], [167], [168] takes an evolutionary approach for program tuning. The Petabricks compiler employs genetic search algorithms to tune algorithmic choices.…”

Section: B Optimise Parallel Programsmentioning

confidence: 99%

Machine Learning in Compiler Optimization

2018

View full text Add to dashboard Cite

Abstract-In the last decade, machine learning based compilation has moved from an an obscure research niche to a mainstream activity. In this article, we describe the relationship between machine learning and compiler optimisation and introduce the main concepts of features, models, training and deployment. We then provide a comprehensive survey and provide a road map for the wide variety of different research areas. We conclude with a discussion on open issues in the area and potential research directions. This paper provides both an accessible introduction to the fast moving area of machine learning based compilation and a detailed bibliography of its main achievements.

show abstract

Hyperparameter Tuning in Bandit-Based Adaptive Operator Selection

Cited by 7 publications

References 8 publications

OpenTuner

OpenTuner

Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions

Machine Learning in Compiler Optimization

Contact Info

Product

Resources

About