Limb amputation and chronic phantom limb pain (PLP) are both associated with neural alterations at all levels of the neuraxis. We investigated gray matter volume of 21 upper limb amputees and 14 healthy control subjects. Results demonstrate that amputation is associated with reduced gray matter in areas in the motor cortex representing the amputated limb. Additionally, patients show an increase in gray matter in brain regions that belong to the dorsal and ventral visual stream. We subdivided the patient group into patients with medium to high PLP (HPLP; N = 11) and those with slight PLP (SPLP; N = 10). HPLP patients showed reduced gray matter in brain areas involved in pain processing. SPLP patients showed a significant gray matter increase in regions of the visual stream. Results indicate that all patients may have an enhanced need for visual control to compensate the lack of sensory feedback of the missing limb. As we found these alterations primarily in the SPLP patient group, successful compensation may have an impact on PLP development. Therefore, we hypothesize that visual adaptation mechanisms may compensate for the lack of sensorimotor feedback and may therefore function as a protection mechanism against high PLP development.