We evaluate the dependence of system performance on the roll-off factor, α, of a Nyquist pulse in a single-channel 1.28 Tbit/s-525 km transmission both experimentally and analytically. Low α values are preferable in terms of spectral efficiency and tolerance to chromatic dispersion and polarization-mode dispersion, while a strong overlap with neighboring symbols results in larger nonlinear impairments. On the other hand, a Nyquist pulse with high α values also suffers from nonlinearity due to higher peak power. As a result, we found experimentally that the optimum α value is 0.4~0.6, which agrees well with the analysis.