Magnetic Resonance Imaging (MRI) is a significant technique used to diagnose brain abnormalities at early stages. This paper proposes a novel method to classify brain abnormalities (tumor and stroke) in MRI images using a hybridized machine learning algorithm. The proposed methodology includes feature extraction (texture, intensity, and shape), feature selection, and classification. The texture features are extracted by intending a neoteric directional-based quantized extrema pattern. The intensity features are extracted by proposing the clustering-based wavelet transform. The shape-based extraction is preformed using conventional shape descriptors. Maximum A Priori (MAP) based firefly algorithm is proposed for feature selection. Finally, hybridized support vector-based random forest classifier is used for the classification. The MRI brain tumor and stroke images are detected and categorized into four classes which are a high-grade tumor, a low-grade tumor, an acute stroke, and a sub-acute stroke. Besides, three different regions are identified in tumor detection such as edema, and tumor (necrotic and non-enhancing) region. The accuracy of the proposed method is analyzed using various performance metrics in comparison with the few state-of-the-art classification methods. The proposed methodology successfully achieves a reliable accuracy of 88.3% for classifying brain tumor cases and 99.2% for brain stroke classification. The best F-score of 0.91 and the least FPR of 0.06 are attained while considering brain tumor classification against the proposed HSVFC. Likewise, HSVFC has 0.99 as the best F-score and a 0.0 FPR in the case of brain stroke classification. The experimental analysis offers a maximum mean accuracy of different classifiers for categorizing MRI brain tumor are 76.