The goal of this paper is to study a well-performed coded modulation structure in visible light communication (VLC) systems equipped with red, green, and blue (RGB) light-emitting diodes (LEDs) to provide tools for performance analysis and to give guidelines for practical designs. Specifically, the protograph-based low-density parity-check (P-LDPC) codes and the optimized generalized variable degree matched mapping interleavers are applied to VLC systems for the first time. To theoretically analyze the system under concern, we utilize the protograph-based extrinsic information transfer (PEXIT) methodology that can accurately predict error performance. Furthermore, by leveraging this powerful analysis tool, we propose optimizing a VLC system from three perspectives, i.e., the interleaver design, the optimization of the power mixing ratio of RGB color lights, and the selection of coding schemes. Both PEXIT analyses and simulations verify that the performance of a VLC system can be greatly improved by optimizing these three components.Index Terms: Visible light communication (VLC), Red, green, and blue light-emitting diode (RGB LED), Low-density parity-check (LDPC), Protograph-based extrinsic information transfer (PEXIT).