Dust devil is an important part of the Martian climate system, which can help us better understand scientific questions of the climate, surface-atmosphere interactions, aeolian processes, and regolith on Mars. Therefore, automatic detection of dust devils from Mars Orbiter images is becoming increasingly important for the scientific study and the planning of future robotic and manned missions. To improve the generalization, detection efficiency and accuracy of traditional approach in automatically detecting dust devils, we made several modifications to the faster region-based convolution neural network (Faster R-CNN). Based on the characteristics of the dust devil, we proposed a Martian Dust Devil Detection Network (MDDD Net). The network uses the feature pyramid network (FPN) to obtain a feature fusion map with rich location information and semantic information. The k-means++ algorithm is used to design reasonable anchor boxes to adapt to vary sized dust devils. Region of interest align (RoI Align) unit is introduced to eliminate the mapping deviation between the feature map and the original image. Finally, the soft non-maximum suppression (Soft-NMS) algorithm is used to complete the screening of bounding box. It can reduce missing detections caused by the overlapping between adjacent dust devil bounding boxes in the same image. The average precision and recall of MDDD Net on the dust devil dataset built in this paper reaches 90.1% and 96.5%, respectively.