As a traditional Chinese flower, chrysanthemum has the advantages of long flowering period, many varieties, wide application, and bright colors. However, different varieties of chrysanthemums require different growth environments and have different ornamental and medicinal values. Therefore, the fine-grained and accurate identification of chrysanthemum varieties has important guiding significance for the breeding and cultivation of chrysanthemums. Finegrained image recognition is a more detailed subclassification of coarse-grained large categories. Due to the subtle interclass differences between sub-categories, compared with general image recognition and classification, it is necessary to use a deeper and more complex network structure to extract more complex and higher-order features of images to improve the the accuracy of fine-grained classification of chrysanthemum images. However, the complexity of the network structure will lead to a large number of parameters of the convolutional neural network model, long training time and weak generalization ability. In order to solve the above problems, this study proposes an improved multi-scale, multi-parallel convolutional neural network, VGG-Inception, based on the traditional VGG-16 convolutional neural network. This model increases the parallel network structure and widens the network while ensuring the depth of the network. VGG-Inception uses the global pooling layer to replace the full connection. While ensuring the accuracy of model classification, the number of model parameters is reduced to 9% of the original VGG-16. At the same time, the auxiliary classifier is used to conduct the gradient forward, avoid the gradient disappearance, and improve the generalization ability of the model.