Abstract-In this paper, we present a robust image alignment algorithm based on matching of relative gradient maps. This algorithm consists of two stages; namely, a learning-based approximate pattern search and an iterative energy-minimization procedure for matching relative image gradient. The first stage finds some candidate poses of the pattern from the image through a fast nearestneighbor search of the best match of the relative gradient features computed from training database of feature vectors, which are obtained from the synthesis of the geometrically transformed template image with the transformation parameters uniformly sampled from a given transformation parameter space. Subsequently, the candidate poses are further verified and refined by matching the relative gradient images through an iterative energy-minimization procedure. This approach based on the matching of relative gradients is robust against nonuniform illumination variations. Experimental results on both simulated and real images are shown to demonstrate superior efficiency and robustness of the proposed algorithm over the conventional normalized correlation method.Index Terms-Energy minimization, illumination variations, image alignment, image matching, industrial inspection, nearestneighbor search, robust image matching.