Eggplant is a popular vegetable crop. Eggplant yields can be affected by various diseases. Automatic detection and recognition of diseases is an important step toward improving crop yields. In this paper, we used a two-stream deep fusion architecture, employing CNN-SVM and CNN-Softmax pipelines, along with an inference model to infer the disease classes. A dataset of 2284 images was sourced from primary (using a consumer RGB camera) and secondary sources (the internet). The dataset contained images of nine eggplant diseases. Experimental results show that the proposed method achieved better accuracy and lower false-positive results compared to other deep learning methods (such as VGG16, Inception V3, VGG 19, MobileNet, NasNetMobile, and ResNet50).