Most comets are volatile-rich bodies that have recently entered the inner solar system following long-term storage in the Kuiper belt and the Oort cloud reservoirs. These reservoirs feed several distinct, short-lived "small body" populations. Here, we present new measurements of the optical colors of cometary and cometrelated bodies including long-period (Oort cloud) comets, Damocloids (probable inactive nuclei of long-period comets) and Centaurs (recent escapees from the Kuiper belt and precursors to the Jupiter family comets). We combine the new measurements with published data on short-period comets, Jovian Trojans and Kuiper belt objects to examine the color systematics of the comet-related populations. We find that the mean optical colors of the dust in short-period and long-period comets are identical within the uncertainties of measurement, as are the colors of the dust and of the underlying nuclei. These populations show no evidence for scattering by optically-small particles or for compositional gradients, even at the largest distances from the Sun, and no evidence for ultrared matter. Consistent with earlier work, ultrared surfaces are common in the Kuiper belt and on the Centaurs, but not in other small body populations, suggesting that this material is hidden or destroyed upon entry to the inner solar system. The onset of activity in the Centaurs and the disappearance of the ultrared matter in this population begin at about the same perihelion distance (∼10 AU), suggesting that the two are related. Blanketing of primordial surface materials by the fallback of sub-orbital ejecta, for which we calculate a very short timescale, is the likely mechanism. The same process should operate on any mass-losing body, explaining the absence of ultrared surface material in the entire comet population.