Timely detection of pests play a major role in agriculture. There exist many pest identification systems, but almost all of them suffer from the misclassification due to lighting, background clutter, heterogeneous capturing devices as well as the pest being partially visible or in the different orientation. This misclassification may cause tremendous yield loss. To mitigate this situation, we proposed an architecture to provide high classification accuracy under the aforementioned conditions using morphology and skeletonization along with neural networks as classifiers. We have considered the crop rice as a use case as it is the staple food grain of almost the entire population of India. The amount of pesticides used is highest in rice as compared to all other food grains. This paper offers a robust technique to identify the pests in rice crops. The performance of the proposed architecture is tested with an image dataset, and the experimental results reveal that our proposed approach provides better classification accuracy than the existing pest detection approaches in the literature. Furthermore, the experimental results also provide the performance comparison among the popular classifiers.