Handwritten character and numeral recognition have gained interest in the research community as part of the big picture of Machine Learning. Writer independent recognition systems are still in the working and the research is geared towards an optimized technique that can achieve this. In this paper, we propose a numeral recognition system that forms fuzzy sets of the features extracted using modified structural features for English, Arabic, Persian, and Devanagari Numerals. The structural features extract the geometrical primitives that distinguish each image. After the feature extraction phase, the results are input into a classifier, we test two different classifiers namely Neural Network and Naïve Base. To further enhance the recognition process with low overhead the erroneously recognized numerals (confusion matrix) are processed through the fuzzy set-based decision mechanism to enhance the numeral recognition process. Results indicate that recognition is enhanced by applying the fuzzy set-based decision mechanism for both classifer.