Automatic human age estimation has attracted a great deal of interest in the past few years. Although many advancements have been made by researchers, there are still many challenges: such as age estimation across different image acquisition methods, different expressions, gender and races. The influence due to race and gender seems to be the most common issue, because collecting a large amount of face images with comprehensive racial diversities seems impractical. The performance will degrade when estimating face images of races that differ from the training set. In this work, we present a new scheme to mitigate the influences of race and gender in the problem of age estimation. Our system will contribute a robust solution to solve the problem of age estimation across races and genders. This study is essential for developing a practical age estimation system (with mixture of races and gender.) To evaluate the performance of the proposed algorithm, we run comprehensive experiments on one widely used big database -MORPH-II, which contains more than 55, 000 images. On an average, an improvement of more than 20% has been achieved using the proposed scheme.