Groundnut is an important oilseed crop in the world, and India is the second-largest producer of groundnuts. This crop is prone to attack by numerous diseases which is one of the most important factors contributing to the loss of productivity and degradation in the quality; both of these finally result in a low agricultural economy. Therefore, it is necessary to find better and more reliable automation solutions to recognize groundnut leaf diseases. In this paper, a deep learning based model with progressive resizing is proposed for groundnut leaf disease recognition and classification tasks. Five major categories of groundnut leaf diseases namely leaf spot, armyworms effect, wilts, yellow leaf, and healthy leaf are considered. The proposed model was trained with and without progressive resizing while it was validated using cross-entropy loss. The first of its kind dataset used for training and validation purposes was manually created from the Saurashtra region of Gujarat state of India. The created dataset was imbalanced in terms of a different number of samples for each category. To handle the imbalanced dataset problem, the extended focal loss function was used. To evaluate the performance of the proposed model, different performance measures including precision, sensitivity, F1-score, and accuracy were applied. The proposed model achieved state-of-the-art accuracy of 96.12%. The model with progressive resizing performed better than the traditional core neural network-based model built on cross-entropy loss.