Indirect far field (IFF) over‐the‐air (OTA) test could be one of the most applicable test methods for 5G mmWave devices. The compact antenna test range (CATR) method has been introduced to the OTA test by 3GPP. Conventional metallic parabolic reflectors without edge treatment can lead to serious edge diffraction which results in high noise and low accuracy. The serrated reflector improves edge diffraction but performs poorly at low frequencies. Reflector antenna with a resistance frequency selection surface (RFSS) frame is proposed for 5G OTA CATR measurement at millimeter frequency, which collimates spherical incident waves into quasi‐plane waves in a relatively short distance, thus forming a quiet zone (QZ) in which large devices under test can be measured in small chamber. In this paper, the reflector with RFSS is investigated through the numerical and experimental methods, it is found that serrated RFSS frame with low edge diffraction could achieve the excellent QZ, which could be helpful in miniaturing CATR chambers for 5G OTA measurement.