With a broad usage of digital imaging data in hospitals, the medical image repository size is growing in a very rapid manner. This creates difficulty in querying and managing these vast databases that in turn leads to the requirement of Content-Based Medical Image Retrieval (CBMIR) systems. The CBMIR is considered as the major ambiguous and challenging task for reducing the semantic gap among the human queries and images in the datasets having more information content.The main intention of this paper is to enhance medical image retrieval and classification using the improved pattern descriptor and deep learning. For the medical image retrieval, this paper develops the Optimized Local Directional Weber Pattern with a multi-objective similarity function. This similarity function focuses on the measures like Structural Similarity Index Measure, Peak Signal-to-Noise Ratio, Mean Squared Error MSE, and correlation. Likewise, the development of an improved Faster-Region Convolutional Neural Network (Faster-RCNN) is adopted in the classification phase. Both the retrieval and classification are enhanced by the Modified Wind Speed-based Deer Hunting Optimization Algorithm. Considering classification and retrieval tasks, the experiments on benchmark datasets reveal