A key issue with palm vein images is that slight movements of fingers and the thumb or changes in the hand pose can stretch the skin in different areas and alter the vein patterns. This can produce palm vein images with an infinite number of variations for a given subject. This paper presents a novel filtering method for SIFT-based feature matching referred to as the Mean and Median Distance (MMD) Filter, which checks the difference of keypoint coordinates and calculates the mean and the median in each direction in order to filter out the incorrect matches. Experiments conducted on the 850nm subset of the CASIA dataset show that the proposed MMD filter can maintain correct points and reduce false positives that were detected by other filtering methods. Comparison against existing SIFT-based palm vein recognition systems demonstrates that the proposed MMD filter produces excellent performance recording lower Equal Error Rate (EER) values.