vDNN: Virtualized deep neural networks for scalable, memory-efficient neural network design

Abstract

The most widely used machine learning frameworks require users to carefully tune their memory usage so that the deep neural network (DNN) fits into the DRAM capacity of a GPU. This restriction hampers a researcher's flexibility to study different machine learning algorithms, forcing them to either use a less desirable network architecture or parallelize the processing across multiple GPUs. We propose a runtime memory manager that virtualizes the memory usage of DNNs such that both GPU and CPU memory can simultaneously be utilized for training larger DNNs. Our virtualized DNN (vDNN) reduces the average GPU memory usage of AlexNet by up to 89%, OverFeat by 91%, and GoogLeNet by 95%, a significant reduction in memory requirements of DNNs. Similar experiments on VGG-16, one of the deepest and memory hungry DNNs to date, demonstrate the memory-efficiency of our proposal. vDNN enables VGG-16 with batch size 256 (requiring 28 GB of memory) to be trained on a single NVIDIA Titan X GPU card containing 12 GB of memory, with 18% performance loss compared to a hypothetical, oracular GPU with enough memory to hold the entire DNN.

PREMA: A Predictive Multi-Task Scheduling Algorithm For Preemptible Neural Processing Units

“…As such, the (GBs of) NPU local memory will be large enough to preserve tens of preempted task's context state. If the multiple checkpointed state oversubscribes NPU memory, the approach taken by Rhu et al [39] can similarly be employed to handle memory oversubscription via copying overflowing data to the CPU memory. Concretely, when the runtime observes that NPU memory usage is nearing its limit, the DMA unit can proactively migrate some of the checkpointed state from NPU to CPU memory while the inference request is being serviced to hide migration overhead.…”

Beyond the Memory Wall: A Case for Memory-Centric HPC System for Deep Learning

“…Memory-overlaying for DNN Virtual Memory. We implemented the runtime memory management policy as described in [9], [30], [10], [52], which leverages the network DAG to analyze inter-layer data dependency to schedule memory-overlaying operations for virtual memory. Under our implementation, the device memory is utilized as an application-level cache with respect to the host memory.…”

“…memory usage of DNNs [9], [10], [13], [14], [15], [16] have proposed to utilize both host and device memory concurrently for allocating data structures for DNN training. By leveraging the user-level DNN topology graph as means to extract a compile-time data dependency information (which is encapsulated as a direct acyclic graph (DAG) data structure) of the memory-hungry data structures, e.g., feature maps (X) and/or weights (W), DNN virtual memory can leverage this data dependency information to derive the DNN data reuse distance to schedule performance-aware data copy operations via memory-overlaying across host and device memory via PCIe [27], [28], [29].…”

“…trained [9], [10], [11], [12]. Increasing the capacity of these on-package stacked DRAM however is challenging due to wireability of the silicon interposers, chip pinout required to drive the added DRAM stacks, and technology limits on how many DRAM stacks you can vertically integrate.…”

High Resolution Medical Image Segmentation Using Data-Swapping Method

“…The first approach involves using the data-swapping method which is proposed in this paper. M. N. Rhu et al [7] and Meng et al [8] also used this approach. They used popular neural networks such as ResNet50 for evaluation and basically focused on the increase in batch size.…”