In this paper, a novel handwritten digit recognition system is proposed. The system consist of feature extraction, feature selection and classification stages. The features of digits are extracted by using the moment-based and structural-based methods. For the moment-based method, wavelet-based two-dimensional scaling moments (2-DSMs), which have uniquely different angular divisions of polar form, are considered. The structural-based features including profiles, intersections of horizontal and vertical straight lines, concavity, number and location of holes are used. In the feature selection stage, Fisher's linear discriminant analysis is used to obtain the discriminative features. The feature selection is performed to improve not only the processing time but also recognition rates. In the classification stage, the digits are classified by neuro-fuzzy classifiers (NFCs). A three-stage cascade NFCs with rejection strategy is used in the system to improve the misclassification rate for the handwritten digit recognition task. The experiments are performed on the MNIST and USPS handwritten digit databases. The high correct classification rates of 98.72 % for MNIST and 97.21 % for USPS are attained by using only one hundred robust hybrid features and cascade NFCs. The experiments showed that the proposed system yields better results among those systems that use only moment-based features.