The origin of a 1/f noise contribution in the long-term carrier-envelope phase (CEP) measurements of mode-locked lasers is discussed. Using two different collinear interferometers for the out-of-loop characterization of feed-forward stabilized Ti:sapphire oscillators, we suppress a possible technical origin of 1/f noise to the extent possible. Both measurements indicate a lower limit of CEP frequency noise of 1 mHz/√Hz. Investigating several possible origins of this noise floor, we find a good agreement with a quantum noise mechanism that is directly induced by intracavity-amplified spontaneous emission (ASE). These findings enable direct access to ASE noise in short-pulse oscillators, which is very hard to characterize via repetition rate fluctuations. Finally, we discuss the possible consequences for frequency-comb-based timekeeping and frequency metrology, as well as for attosecond science.