This paper presents a detailed study of the acoustic response of artificially manufactured notches, which are established reference defects in common standards for ultrasonic testing (UT). Measurements were performed with standard tubes manufactured for the commissioning of automated UT machines. Included reference notches are at the inner and outer tube surface and have different lengths and orientations (longitudinal, oblique and transverse). From the acoustic reflection profile, i.e. the measured defect amplitude along the notch axis, the effective defect length and the defect homogeneity are introduced as measured quantities of the reference defect quality. The large data sample of 320 evaluated notch profiles gives insight to the statistics of the achieved reference defect quality. Key approach of the analysis is the correlation between both measured quantities. It reveals a universal behaviour, which can be described by a sigmoidal function converging for high quality defects towards the theoretical limits. Based on the obtained parametrization it is possible to define criterions for the required defect quality and the test speed as function of the specified repeatability. This solves the current dilemma how to distinguish between variations due to the reference quality from those related to the testing machine. Furthermore, the presented measurement procedure can be used for a reference certification, which complements existing standards for the ultrasonic system and the probes.