The energy crisis and environmental problems have attracted global attention, thus the photovoltaic (PV) power generation technology comes to people’s mind. The application of unmanned aerial vehicle (UAV) inspection technology can overcome the disadvantages of large scale and high risk of this project. The application of unmanned aerial vehicle (UAV) infrared detection technology in PV power generation can not only improve work efficiency, but also have high economic benefits. This paper based on U-Net network and HSV space, proposes a method of PV infrared image segmentation and location detection of hot spots, which is used to detect and analyze the shielding of PV panels. Firstly, the main PV modules are automatically split from the different infrared image background based on U-Net. In order to quickly locate the defection location, the mask image is multiplied by the original image and then converted to HSV. The discriminant of bright spot features is introduced, and the discriminant mechanism is summarized according to the experiment, and the formation reason is analyzed. The experiment result shows that the method is not affected by the infrared image under the different background, provides data for the maintenance of power station and improves the detection accuracy. The accuracy rate of analyzing the causes of defects is 92.5%.