A systematic coarse graining of microscopic atomic displacements generates a local elastic deformation tensor D as well as a positive definite scalar χ measuring nonaffinity, i.e., the extent to which the displacements are not representable as affine deformations of a reference crystal. We perform an exact calculation of the statistics of χ and D and their spatial correlations for solids at low temperatures, within a harmonic approximation and in one and two dimensions. We obtain the joint distribution P(χ,D) and the two-point spatial correlation functions for χ and D. We show that nonaffine and affine deformations are coupled even in a harmonic solid, with a strength that depends on the size of the coarse-graining volume Ω and dimensionality. As a corollary to our work, we identify the field h(χ) conjugate to χ and show that this field may be tuned to produce a transition to a state where the ensemble average <χ> and the correlation length of χ diverge. Our work should be useful as a template for understanding nonaffine displacements in realistic systems with or without disorder and as a means for developing computational tools for studying the effects of nonaffine displacements in melting, plastic flow, and the glass transition.