Depression and early life adversity (ELA) are associated with aberrant resting state functional connectivity (FC) of the default mode (DMN), salience (SN), and central executive networks (CEN). However, the specific and differential associations of depression and ELA with FC of these networks remain unclear. Applying a dimensional approach, here we analyzed associations of FC between major nodes of the DMN, SN, and CEN with severity of depressive symptoms and ELA defined as childhood abuse and neglect in a sample of 83 healthy and depressed subjects. Depressive symptoms were linked to increased FC within the SN and decreased FC of the SN with the DMN and CEN. Childhood abuse was associated with increased FC within the SN, whereas childhood neglect was associated with decreased FC within the SN and increased FC between the SN and the DMN. Our study thus provides evidence for differential associations of depressive symptoms and ELA with resting state FC and contributes to a clarification of previously contradictory findings. Specific FC abnormalities may underlie specific cognitive and emotional impairments. Future research should link specific clinical symptoms resulting from ELA to FC patterns thereby characterizing depression subtypes with specific neurobiological signatures.