Due to numerous difficulties, including the variation in face shapes between individuals, the challenge of recognizing dynamic facial attributes, the poor quality of digital images, etc., detecting human emotion depending on facial expression is difficult for the computer vision community. Thus, in this study, we propose an approach for emotion recognition depending on facial expression using histogram of oriented gradients and convolution neural network (HOG-CNN). The HOG-CNN composed of three stages, median filter, HOG, and CNN. The first stage is preprocessing using median filter. The second stage is feature extraction using HOG. The third stage is classification using CNN. The proposed method was tested and evaluated on the UMD face database. The system attained a high performance with a mean average accuracy of 98.07%, average precision of 94.78%, and average recall of 97.15%.