Over the past decade, there has been unprecedented advancements in the field of computer vision by adopting AI-based solutions. In particular, cutting edge computer vision technology based on deep-learning approaches has been deployed with an extraordinary degree of success. The ability to extract semantic concepts from continuous processing of video stream in real-time has led to the investigation of such solutions to enhance the operational security of critical infrastructure against intruders. Despite the success of computer vision technologies validated in a laboratory environment, there still exists several challenges that limit the deployment of these solutions in operational environment. Addressing these challenges, the paper presents a framework that integrates three main computer vision technologies namely (i) person detection; (ii) person re-identification and (iii) face recognition to enhance the operational security of critical infrastructure perimeter. The novelty of the proposed framework relies on the integration of key technical innovations that satisfies the operational requirements of critical infrastructure in using computer vision technologies. One such requirement relates to data privacy and citizen rights, following the implementation of General Data Protection Regulation across Europe for the successful adoption of video surveillance for infrastructure security. The video analytics solution proposed in the paper integrates privacy preserving technologies, high-level rule engine for threat identification and a knowledge model for escalating threat categorises to human operator. The various components of the proposed framework has been validated using commercially available graphical processing units for detecting intruders. The performance o the proposed framework has been evaluated in operational environments of the critical infrastructure. An overall accuracy of 97% is observed in generating alerts against malicious intruders.