Abstract-The temporal drift due to the polarization noise in an ultra-low temporal drift photonic phase reference distribution system is analyzed. The temporal drift is shown to be associated with the polarization changes, the differential group delay of the subsequent parts, and, the state of polarization dispersion of the photonic local oscillator signal. Such temporal drift cannot be corrected by Line Length Correction schemes. As an example, polarization changes from a fiber stretcher are presented, discussed, and identified as one of the major contributors to the system temporal drift. A numerical model is developed. The magnitude and statistical distribution (based on random coupling) of the temporal drift excited by the polarization change of the fiber stretcher are investigated. Experimental work demonstrates that reduction in the polarization change successfully reduces the temporal drift, as predicted by the theory and simulation.