Cadmium (Cd) is a toxic element, and rice is known to be a leading source of dietary Cd for people who consume rice as their main caloric resource. Hybrid rice has dominated rice production in southern China and has been adopted worldwide. The characteristics of high yield heterosis of rice hybrids makes the public think intuitively that the hybrid rice accumulates more Cd in grain than do inbred cultivars. A detailed understanding of the genetic basis of grain Cd accumulation in hybrids and developing Cd-safe rice are one of the top priorities for hybrid rice breeders at present. In this study, we investigated genetic diversity and grain Cd levels in 617 elite rice hybrids collected from the middle and lower Yangtze River Valley in China and 68 inbred cultivars from around the world. We found that there are large variations in grain Cd accumulation in both the hybrids and their inbred counterparts. However, we found grain Cd levels in the rice hybrids to be similar to the levels in indica rice inbreds, suggesting that the hybrids do not accumulate more Cd than do the inbred rice cultivars. Further analysis revealed that the high heritability of Cd accumulation in the grain and the single indica population structure increases the risk of Cd over-accumulation in hybrid rice. The genetic effects of Cd-related QTLs, which have been identified in related Cd-QTL mapping studies, were also determined in the hybrid rice population. Four QTLs were identified as being associated with the variation in grain Cd levels; three of these loci exhibited obvious indica-japonica differentiations. Our study will provide a better understanding of grain Cd accumulations in hybrid rice, and pave the way toward effective breeding for high-yielding, low grain-Cd hybrids in the future.