In recent years, video event prediction and classification is considered a hot research topic among researchers, due to its social influence and its extensive real-time applications. In this research, a video event classification system is proposed to predict and categorize various day-to-day events. This research proposed an improved pelican optimization algorithm (IPOA) to optimize the hyper-parameters which aids in better classification accuracy. The input data is gathered from three well-known datasets such as University of Central Florida101 (UCF101), Human Motion Database 51 (HMDB51) and columbia consumer video (CCV) dataset. The raw data is preprocessed using the data normalization technique and the DenseNet 201 model is used to extract the features from the pre-processed output. The extracted features are fed into the stage of feature selection using the improved grey wolf optimization (IGWO) algorithm. The hyper-parameters of the selected features are optimized using the proposed IPOA algorithm and classification takes place using bidirectional long short term memory (Bi-LSTM) classifier. The experimental results show that the proposed IPOA with Bi-LSTM achieved a better classification accuracy of 93. 91%, 94.19%, and 90.19% for datasets such as UCF101, HMDB51 and CCV respectively which is comparatively higher than the existing techniques such as improved residual convolutional neural network (CNN), Bi-LSTM CNN, twostream 3D CNN model and gait event detection system.