According to a classification of central nervous system tumors by the World Health Organization, diffuse gliomas are classified into grade 2, 3, and 4 gliomas in accordance with their aggressiveness. To quantitatively evaluate a tumor’s malignancy from brain magnetic resonance imaging, this study proposed a computer-aided diagnosis (CAD) system based on a deep convolutional neural network (DCNN). Gliomas from a multi-center database (The Cancer Imaging Archive) composed of a total of 30 grade 2, 43 grade 3, and 57 grade 4 gliomas were used for the training and evaluation of the proposed CAD. Using transfer learning to fine-tune AlexNet, a DCNN, its internal layers, and parameters trained from a million images were transferred to learn how to differentiate the acquired gliomas. Data augmentation was also implemented to increase possible spatial and geometric variations for a better training model. The transferred DCNN achieved an accuracy of 97.9% with a standard deviation of ±1% and an area under the receiver operation characteristics curve (Az) of 0.9991 ± 0, which were superior to handcrafted image features, the DCNN without pretrained features, which only achieved a mean accuracy of 61.42% with a standard deviation of ±7% and a mean Az of 0.8222 ± 0.07, and the DCNN without data augmentation, which was the worst with a mean accuracy of 59.85% with a standard deviation ±16% and a mean Az of 0.7896 ± 0.18. The DCNN with pretrained features and data augmentation can accurately and efficiently classify grade 2, 3, and 4 gliomas. The high accuracy is promising in providing diagnostic suggestions to radiologists in the clinic.