MAPLE: Microprocessor A Priori for Latency Estimation

Abstract

Modern deep neural networks must demonstrate state-of-the-art accuracy while exhibiting low latency and energy consumption. As such, neural architecture search (NAS) algorithms take these two constraints into account when generating a new architecture. However, efficiency metrics such as latency are typically hardware dependent requiring the NAS algorithm to either measure or predict the architecture latency. Measuring the latency of every evaluated architecture adds a significant amount of time to the NAS process. Here we propose Microprocessor A Priori for Latency Estimation MAPLE that does not rely on transfer learning or domain adaptation but instead generalizes to new hardware by incorporating a prior hardware characteristics during training. MAPLE takes advantage of a novel quantitative strategy to characterize the underlying microprocessor by measuring relevant hardware performance metrics, yielding a fine-grained and expressive hardware descriptor. Moreover, the proposed MAPLE benefits from the tightly coupled I/O between the CPU and GPU and their dependency to predict DNN latency on GPUs while measuring microprocessor performance hardware counters from the CPU feeding the GPU hardware. Through this quantitative strategy as the hardware descriptor, MAPLE can generalize to new hardware via a few shot adaptation strategy where with as few as 3 samples it exhibits a 3% improvement over state-of-the-art methods requiring as much as 10 samples. Experimental results showed that, increasing the few shot adaptation samples to 10 improves the accuracy significantly over the state-of-the-art methods by 12%. Furthermore, it was demonstrated that MAPLE exhibiting 8-10% better accuracy, on average, compared to relevant baselines at any number of adaptation samples. The proposed method provides a versatile and practical latency prediction methodology inferring DNN run-time on multiple hardware devices while not imposing any significant overhead for sample collection.Preprint. Under review.

MAPLE-X: Latency Prediction with Explicit Microprocessor Prior Knowledge

“…In this section we evaluate and compare MAPLE-X with MAPLE [1] and HELP [4]. We adopt a one-device-leave-out approach and form a training pool of five devices listed in Section 3.1 and use the sixth device for testing.…”

“…To evaluate our methodology, we use the same dataset as MAPLE which we refer to as NASBench-X [1]. NASBench-X is based on NASBench-201, a cell-based convolutional neural network NAS dataset and includes a total of 15,625 architectures.…”

“…A recently introduced technique known as MAPLE [1] employs hardware performance counters to characterize how different devices behave under specific workloads. MAPLE subsequently measures a small number of samples (i.e., 3 to 10) from the target hardware and augments the training set prior to optimizing a hardware-aware regression model.…”

Inference Latency Prediction at the Edge

“…As an alternative to direct measurements, existing approaches for evaluating the efficiency of a neural architecture can be categorized as those using: (1) Proxy metrics [50,65] (e.g., FLOPs), which are usually platform-independent and cannot accurately reflect the actual performance due to the diversity of platforms [39,51]. (2) Lookup tables [7,11,56], which are collected for pre-defined building blocks in the search space, but cannot cover every possible configuration in a potentially huge search space and require comprehensive measurements on each platform. (3) Prediction models [2,8,12], which broadly rely on machine learning techniques (e.g., MLPs) and have the potential to predict the performance of any configuration in the search space.…”

“…(2) Lookup tables [7,11,56], which are collected for pre-defined building blocks in the search space, but cannot cover every possible configuration in a potentially huge search space and require comprehensive measurements on each platform. (3) Prediction models [2,8,12], which broadly rely on machine learning techniques (e.g., MLPs) and have the potential to predict the performance of any configuration in the search space. However, it is difficult to build accurate prediction models for efficiency metrics on mobile devices due to the following challenges.…”