In order to meet the transmission needs of WDM-PON system, a WDM-PON full-duplex system based on reflective semiconductor optical amplifier (RSOA) remodulation was constructed in this paper. The downlink transmission rate of the system was 40Gb/s, and the downlink adopted three modulation formats, namely carrier suppression return to zero (CSRZ), optical duobinary (ODB) and quadrary pulse amplitude modulation (PAM4); the uplink transmission rate of the system was 10Gb/s, and the uplink adopted NRZ code. Through the simulation, the transmission performance of the system under these three modulation formats was compared and analysed?and the system uplink/downlink eye diagram and Q-value graph were obtained. The results show that the PAM4 signal has the strongest anti-dispersion capability and the ODB has the highest anti-nonlinear tolerance.