Electron-defect (e-d) interactions govern charge carrier dynamics at low temperature, where they limit the carrier mobility and give rise to phenomena of broad relevance in condensed matter physics. Ab initio calculations of e-d interactions are still in their infancy, mainly because they require large supercells and computationally expensive workflows. Here we develop an efficient ab initio approach for computing elastic e-d interactions, their associated e-d relaxation times (RTs), and the lowtemperature defect-limited carrier mobility. The method is applied to silicon with simple neutral defects, such as vacancies and interstitials. Contrary to conventional wisdom, the computed e-d RTs depend strongly on carrier energy and defect type, and the defect-limited mobility is temperature dependent. These results highlight the shortcomings of widely employed heuristic models of e-d interactions in materials. Our method opens new avenues for studying e-d scattering and lowtemperature charge transport from first principles.