Photon-assisted millimeter wave system can improve real-time transmission performance of terahertz radio signals. And wireless communication is the uppermost form in nowadays communication industry. However, it is necessary to get over the multipath effect subjected to signal during wireless channel transmission. Firstly, the impact of multipath effect and the cause of frequency selective fading are introduced. Then the principle of photon-assisted mm-wave wireless transmission system is described. In this paper, we mainly focus on the frequency selective fading caused by delay extension in multipath effects, as well as the ISI and constellation distortion which the frequency selective fading generates. Here we have simulated the Rician fading channel by MATLAB, and established QPSK, 16-APSK & 16-QAM modulation techniques. Next evaluating the performance to resist frequency selective fading of these three modulation format photon-assisted mm-wave system by comparing their BER and constellation distortion. The performance has been concluded based on BER vs. MaxPathDelay and constellations output for QPSK, 16-APSK & 16-QAM over Rician channel is that QPSK is better than 16-APSK and 16-QAM. Because QPSK system over Rician channel at MaxPathDelay=9×10 -10 s, BER=0.025. When MaxPathDelay=15×10 -10 , only the constellation of QPSK system does not occur significant distortion.