Multitone-frequency and quadrature-amplitude modulation (MFQAM) is a combination of multitone frequencyshift keying and quadrature-amplitude modulation (QAM). In this study, the authors consider MFQAM with active subcarriers grouped in balance, where the active subcarriers are divided into a specified number of groups and the same QAM symbol is conveyed by every subcarrier in each group. This grouping may make MFQAM have frequency diversity in itself. They first derive the joint probability density function of the correlator outputs of an MFQAM receiver. They then compute the channel capacity of a coded modulation (CM) system employing MFQAM over Rayleigh fading channels. As a result, they demonstrate that for a moderate code rate between 1/3 and 2/3, its normalised throughput is close to the channel capacity limit in the low signal-to-noise ratio region, contrary to a CM system employing the conventional modulation. They also evaluate the frame error rate (FER) performance of a CM system employing MFQAM when a non-binary error-correcting code is employed. Numerical results demonstrate that the normalised throughput of the CM system computed via the FER shows a similar trend to that obtained via the channel capacity.