In computer vision, face tracking is having wider opportunities for research activities using different background video sequences because of various factors and constraints. Due to the challenges that are increasing day by day, old/existing algorithms are becoming obsolete. There are many powerful algorithms that are limited to certain set of video sequences. In this paper, we are proposing an algorithm that detect and track multiple faces in different background video sequences. Viola-Jones face detection algorithm is used in such a way that, new face/first face need not to be in the starting frame of the selected video sequence. The proposed algorithm successfully detect new face(s) along with existing face(s) by keeping track of the facial data using BRISK feature points. The mean of the old points and new points are calculated based on the area of the facial data. The detected face(s) in further frames undergoes similarity check with existing facial data. If detected facial data and existing facial data mismatches, then the detected facial data is entered into face tracks structure. By using point tracker method, the proposed algorithm track those points that has been set for each of the facial data.