Malaria poses an enormous threat to humanity with ever increasing cases annually. The research in the field of medical is contributing quite a lot in providing methods for premature diagnosis of malaria. Apart from medical research, information technology is also playing a vital role in proposing efficient methods for malaria diagnosis. To minimize the manual interference and boost the diagnosis accuracy, the automated systems are under study lately. An ensemble deep learning scheme is proposed in this paper with the fusion of features obtained by two state-of-the-art pre-trained deep Convolutional Neural Networks (CNN) models ResNet101 and SqueezeNet for the classification of malaria blood smears from red blood cells. A handcrafted feature extractor Local Binary Patterns (LBP) is also implemented along with the fused deep model features to deduce texture features of infected regions within image for better discrimination. Linear Discriminant Analysis (LDA) is employed for feature selection and optimization. At the end, the selected features are classified using a set of classifiers. The proposed Optimized Deep Malaria Classifier (ODMC) model achieved 99.73% accuracy with exceptional time efficiency.