Silicon photonics has enormous potential for ultrahigh-capacity coherent optical transceivers. We demonstrate an IQ modulator using silicon photonic traveling-wave modulators optimized for higher-order quadrature amplitude modulation (QAM). Its optical and RF characteristics are studied thoroughly in simulation and experiment. We propose a system-orientated approach to optimization of the silicon photonic IQ modulator, which minimizes modulator-induced power penalty in a QAM transmission link. We examine the trade-off between modulation efficiency and bandwidth for the optimal combination of modulator length and bias voltage to maximize the clear distance between adjacent constellation points. This optimum depends on baud rate and modulation format, as well as achievable driving voltage swing. Measured results confirm our prediction using the proposed methodology. Without pre-compensating bandwidth limitation of the modulator, net data rates up to 232 Gb/s (70 Gbaud 16-QAM) on single polarization are captured, indicating great potential for 400+ Gb/s dual-polarization transmission.