In the development and application of dispersion-corrected density-functional theory, the effects of basis set incompleteness have been largely mitigated through the use of very large, nearly-complete basis sets. However, the use of such large basis sets makes application of these methods inefficient for large systems. In this work, we examine a series of basis sets, including Pople-style, correlation-consistent, and polarization-consistent bases, for their ability to efficiently and accurately predict non-covalent interactions when used in conjunction with the exchange-hole dipole moment (XDM) dispersion model. We find that the polarization-consistent 2 (pc-2) basis sets, and two modifications thereof with some diffuse functions removed, give performance of comparable quality to that obtained with aug-cc-pVTZ basis sets, while being roughly 12 to 23 times faster computationally. The behavior is explained, in part, by the role of diffuse functions in recovering small density changes in the intermolecular region. The general performance of the modified basis sets is tested by application of XDM to standard intermolecular benchmark sets at, and away from, equilibrium.