In general, a reliable and automatic semiconductor fabrication processes is of great importance to product yield and cost reduction. In the past, we made use of human vision to do wafer die defect detection and classification, which is hindered by the easy fatigue and fuzziness of human eyes and the decision difference between inspectors. In this paper, we develop a vision-based automatic defect classification system. In our defect detection component, we apply the MAD method to align the test image with the reference image. To acquire the binary defect images, we subtract the test image from the reference image, then we convert the difference image into the binary image by setting a threshold. Moreover, we removed the scattering noises by setting a minimum number of connected noisy pixels required. Finally, we extract all defects in the test image in order to perform the defect classification.For defect classification, we revise the ART 1 model, which still can retain its stability and the plasticity dilemma. We have found that ART 1 shows an intolerable shortcoming: whose output is dependent on the ordering of input sequence applied to the ART 1 model. To remedy this disadvantage, we derive the similarity based ART 1 model which can obtain high classification accuracy and independent on the ordering of input patterns applied.Keywords-similarity based ART 1 model; die defect detection; die defect classification; input pattern ordering independence