MRI based on T1 relaxation contrast is increasingly being used to study brain morphology and myelination. Although it provides for excellent distinction between the major tissue types of grey matter, white matter, and CSF, reproducible quantification of T1 relaxation rates is difficult due to the complexity of the contrast mechanism and dependence on experimental details. In this work, we perform simulations and inversion-recovery MRI measurements at 3 T and 7 T to show that substantial measurement variability results from unintended and uncontrolled perturbation of the magnetization of MRI-invisible 1H protons of lipids and macromolecules. This results in bi-exponential relaxation, with a fast component whose relative contribution under practical conditions can reach 20%. This phenomenon can strongly affect apparent relaxation rates, affect contrast between tissue types, and result in contrast variations over the brain. Based on this novel understanding, ways are proposed to minimize this experimental variability and its effect on T1 contrast, quantification accuracy and reproducibility.