Context. We investigate the formation and evolution of comet nuclei and other trans-Neptunian objects (TNOs) in the solar nebula and primordial disk prior to the giant planet orbit instability foreseen by the Nice model. Aims. Our goal is to determine whether most observed comet nuclei are primordial rubble-pile survivors that formed in the solar nebula and young primordial disk or collisional rubble piles formed later in the aftermath of catastrophic disruptions of larger parent bodies. We also propose a concurrent comet and TNO formation scenario that is consistent with observations. Methods. We used observations of comet 67P/Churyumov-Gerasimenko by the ESA Rosetta spacecraft, particularly by the OSIRIS camera system, combined with data from the NASA Stardust sample-return mission to comet 81P/Wild 2 and from meteoritics; we also used existing observations from ground or from spacecraft of irregular satellites of the giant planets, Centaurs, and TNOs. We performed modeling of thermophysics, hydrostatics, orbit evolution, and collision physics. Results. We find that thermal processing due to short-lived radionuclides, combined with collisional processing during accretion in the primordial disk, creates a population of medium-sized bodies that are comparably dense, compacted, strong, heavily depleted in supervolatiles like CO and CO 2 ; they contain little to no amorphous water ice, and have experienced extensive metasomatism and aqueous alteration due to liquid water. Irregular satellites Phoebe and Himalia are potential representatives of this population. Collisional rubble piles inherit these properties from their parents. Contrarily, comet nuclei have low density, high porosity, weak strength, are rich in supervolatiles, may contain amorphous water ice, and do not display convincing evidence of in situ metasomatism or aqueous alteration. We outline a comet formation scenario that starts in the solar nebula and ends in the primordial disk, that reproduces these observed properties, and additionally explains the presence of extensive layering on 67P/Churyumov-Gerasimenko (and on 9P/Tempel 1 observed by Deep Impact), its bi-lobed shape, the extremely slow growth of comet nuclei as evidenced by recent radiometric dating, and the low collision probability that allows primordial nuclei to survive the age of the solar system. Conclusions. We conclude that observed comet nuclei are primordial rubble piles, and not collisional rubble piles. We argue that TNOs formed as a result of streaming instabilities at sizes below ∼400 km and that ∼350 of these grew slowly in a low-mass primordial disk to the size of Triton, Pluto, and Eris, causing little viscous stirring during growth. We thus propose a dynamically cold primordial disk, which prevented mediumsized TNOs from breaking into collisional rubble piles and allowed the survival of primordial rubble-pile comets. We argue that comets formed by hierarchical agglomeration out of material that remained after TNO formation, and that this slow growth was a necessity...