This paper attempts to replace the traditional manual slag offloading of magnesium smelting in Pidgeon process with robotic slag removal. Specifically, the high-temperature infrared dot matrix was used to measure the slag positions indirectly; the faster region-based convolutional neural network (Faster R-CNN) was trained with thermal image of the reduction jar as the dataset; the isothermal image of the reduction jar was plotted based on the slag centers, and adopted to detect the opening direction of the jar and the slag positions. The indirect measurement results show that the actual internal temperature of the jar can be detected accurately through repeated experiments, with an error of less than 10 °C. Finally, the proposed method was verified through a case study on 1,000 images. The results show that our model can correctly identify more than 90% of crude magnesium in the actual jar.