SUMMARYWe first describe in a unified way how to compute the covariance matrix from the gray levels of the image. We then experimentally investigate whether or not the computed covariance matrix actually reflects the accuracy of the feature position by doing subpixel correction using variable template matching. We also test if the accuracy of the homography and the fundamental matrix can really be improved by optimization using the covariance matrix computed from the gray levels.