The transmission performance of three signaling formats in long-haul ultra-dense wavelength division multiplexed systems is investigated numerically. The signaling formats are the single-sideband duobinary-carrier-suppressed return-to-zero (SSB-DCS-RZ), the SSB alternate mark inversion RZ, and the bandwidth-limited phase-shaped binary transmission (BL-PSBT). After optimizing the bandwidths and detunings of the optical multiplexer and demultiplexer, the dispersion map is also optimized, and the dispersion and nonlinearity tolerances of all the signaling formats are assessed.It is shown that higher Q-factor and better tolerance to total residual dispersion are achieved by the BL-PSBT format. However, the SSB-DCS-RZ format has a remarkable improvement (exceeding 38%) of in-line dispersion compensation tolerance with respect to BL-PSBT. Furthermore, the three formats have similar nonlinear threshold.Index Terms-alternate mark inversion (AMI), duobinarycarrier-suppressed (DCS), fiber nonlinearity, phase-shaped binary transmission (PSBT), residual dispersion, ultra-dense wavelength division multiplexing.