Fibroblasts are non-hematopoietic structural cells that define the architecture of organs, support the homeostasis of tissue-resident cells and play key roles in fibrosis, cancer, autoimmunity and wound healing. Recent studies have described fibroblast heterogeneity within individual tissues. However, the field lacks a definition of fibroblasts at single-cell resolution across tissues in healthy and diseased organs. Here, we integrated single-cell RNA transcriptomic data from ~150,000 fibroblast cells derived from 16 steady- and 11 perturbed-state mouse organs into fibroblast atlases. These data revealed two universal fibroblast cell subtypes, marked by expression of Pi16 or Col15a1, in all tissues; it also revealed discrete subsets of five specialized fibroblast subtypes in steady-state tissues and three activated fibroblast subtypes in perturbed or diseased tissues. These subsets were transcriptionally shaped by microenvironmental context rather than tissue-type alone. Inference of fibroblast lineage structure from the murine steady-state and perturbed-state fibroblast atlases suggested that specialized and activated subtypes are developmentally related to universal tissue-resident fibroblasts. Analysis of human samples revealed that fibroblast subtypes found in mice are conserved between species, including universal fibroblasts and activated phenotypes associated with pathogenicity in human cancer, fibrosis, arthritis and inflammation. In sum, a cross-species and pan-tissue approach to transcriptomics at single-cell resolution enabled us to define the organizing principles of the fibroblast lineage in health and disease.