Face recognition is still a very demanding area of research. This problem becomes more challenging in unconstrained environment and in the presence of several variations like pose, illumination, expression, etc. Local descriptors are widely used for this task. The most of the existing local descriptors consider only few immediate local neighbors and not able to utilize the wider local information to make the descriptor more discriminative. The wider local information based descriptors mainly suffer due to the increased dimensionality. In this paper, this problem is solved by encoding the relationship among directional neighbors in an efficient manner. The relationship between the center pixel and the encoded directional neighbors is utilized further to form the proposed local directional relation pattern (LDRP). The descriptor is inherently uniform illumination invariant. The multi-scale mechanism is also adapted to further boost the discriminative ability of the descriptor. The proposed descriptor is evaluated under the image retrieval framework over face databases. Very challenging databases like PaSC, LFW, PubFig, ESSEX, FERET, AT&T, and FaceScrub are used to test the discriminative ability and robustness of LDRP descriptor. Results are also compared with the recent state-of-the-art face descriptors such as LBP, LTP, LDP, LDN, LVP, DCP, LDGP and LGHP. Very promising performance is observed using the proposed descriptor over very appealing face databases as compared to the existing face descriptors. The proposed LDRP descriptor also outperforms the pretrained ImageNet CNN models over large-scale FaceScrub face dataset. Moreover, it also outperforms the deep learning based DLib face descriptor in many scenarios.Unconstrained and robust face recognition is the current demand for the betterment of the quality life. Most of the early days research has been conducted in a very controlled environment, where users have given their facial images in frontal pose, under consistent lighting, without glasses or occlusion, etc. Some researchers also tried to develop the face recognition approaches robust for specific geometric and photometric changes such as pose, illumination, motion blur, etc. [68], [14], [29], [15], [50]. The face recognition approaches are surveyed time to time by many researchers [75], [74], [13]. The face recognition approaches are categorized into three major areas, namely deep learning based face recognition [58], [62], traditional learning based face recognition [6], [67], [33], [41], [40], and hand-crafted feature based face recognition [71], [22], [7]. The deep learning based approaches are being popular due to high performance, but at the cost of increased complexity in terms of the time, computing power and data size. The deep learning based approaches are also biased towards the training data. The main drawback of the traditional learning based descriptors are the dependency over the training database and vocabulary size. The hand-designed local descriptors are very simple from design aspect. ...