We present the derivation of the Casimir-Polder interactions mediated by a massive photon between two neutral systems described in terms of their atomic polarizability tensors. We find a compact expression for the leading term at large distances between the two systems. Our result reduces, in the mass-less photon limit, to the standard Casimir-Polder. We discuss implications of our findings with respect to recent scenarios of physics beyond the standard model such as universal extra dimensions, Randall-Sundrum and scale-invariant models. For each model we compute the correction to the Casimir-Polder interaction in terms of the free parameters.