Measurement of the pulsar timing residuals provides a direct way to detect relic gravitational waves at the frequency f ∼ 1/yr. In this paper, we investigate the constraints on the inflationary parameters, the tensor-to-scalar ratio r and the tensor spectral index nt, by the current and future Pulsar Timing Arrays (PTAs). We find that Five-hundred-meter Aperture Spherical radio Telescope (FAST) in China and the planned Square Kilometer Array (SKA) projects have the fairly strong abilities to test the phantom-like inflationary models. If r = 0.1, FAST could give the constraint on the spectral index nt < 0.56, and SKA gives nt < 0.32. While an observation with the total time T = 20yr, the pulsar noise level σw = 30ns and the monitored pulsar number n = 200, could even constrain nt < 0.07. These are much tighter than those inferred from the current results of Parkers Pulsar Timing Array (PPTA), European Pulsar Timing Array (EPTA) and North American Nanohertz Observatory for Gravitational waves (NANOGrav). Especially, by studying the effects of various observational factors on the sensitivities of PTAs, we found that compared with σw and n, the total observation time T has the most significant effect.