Surface acoustic wave (SAW) sensors for the detection of magnetic fields are currently being studied scientifically in many ways, especially since both their sensitivity as well as their detectivity could be significantly improved by the utilization of shear horizontal surface acoustic waves, i.e., Love waves, instead of Rayleigh waves. By now, low-frequency limits of detection (LOD) below 100 pT/Hz can be achieved. However, the LOD can only be further improved by gaining a deep understanding of the existing sensor-intrinsic noise sources and their impact on the sensor’s overall performance. This paper reports on a comprehensive study of the inherent noise of SAW delay line magnetic field sensors. In addition to the noise, however, the sensitivity is of importance, since both quantities are equally important for the LOD. Following the necessary explanations of the electrical and magnetic sensor properties, a further focus is on the losses within the sensor, since these are closely linked to the noise. The considered parameters are in particular the ambient magnetic bias field and the input power of the sensor. Depending on the sensor’s operating point, various noise mechanisms contribute to f0 white phase noise, f−1 flicker phase noise, and f−2 random walk of phase. Flicker phase noise due to magnetic hysteresis losses, i.e. random fluctuations of the magnetization, is usually dominant under typical operating conditions. Noise characteristics are related to the overall magnetic and magnetic domain behavior. Both calculations and measurements show that the LOD cannot be further improved by increasing the sensitivity. Instead, the losses occurring in the magnetic material need to be decreased.