The problem of increasing the speed of railway transportation and ensuring reliability is associated with constant monitoring of the condition of the railway tracks. The modern track measuring cars are equipped with video cameras and computer equipment for processing the received information. However, manual processing of data by operators in real-time is not possible. The article proposes a deep convolutional neural network for automatically recognizing and classifying defects in rail joints on rail track images. The rail video observation forms the image array during the passage of the track recording car. The formation of classes of rail joints is described. Regular rail joints with connectors, insulating joints, and welded joints are considered. Additional classes are identified, corresponding to various anomalous configurations of rail joints in the images. A modified structure of a pre-trained deep convolutional network is constructed. When preparing training samples, the actual images of rail joints were supplemented with artificial images obtained by affine transformations. The process of training and testing the classifier based on the developed convolutional network is described. The Transfer Learning is used to train the neural network. As a result of the experiments, the accuracy of classifying rail joints and detecting defects was at least 96%.