We present the deepest 100 to 500 μm far-infrared observations obtained with the Herschel Space Observatory as part of the GOODS-Herschel key program, and examine the infrared (IR) 3-500 μm spectral energy distributions (SEDs) of galaxies at 0 < z < 2.5, supplemented by a local reference sample from IRAS, ISO, Spitzer, and AKARI data. We determine the projected star formation densities of local galaxies from their radio and mid-IR continuum sizes. We find that the ratio of total IR luminosity to rest-frame 8 μm luminosity, IR8 (≡L tot IR /L 8 ), follows a Gaussian distribution centered on IR8 = 4 (σ = 1.6) and defines an IR main sequence for star-forming galaxies independent of redshift and luminosity. Outliers from this main sequence produce a tail skewed toward higher values of IR8. This minority population (<20%) is shown to consist of starbursts with compact projected star formation densities. IR8 can be used to separate galaxies with normal and extended modes of star formation from compact starbursts with high-IR8, high projected IR surface brightness (Σ IR > 3 × 10 10 L kpc −2 ) and a high specific star formation rate (i.e., starbursts). The rest-frame, UV-2700 Å size of these distant starbursts is typically half that of main sequence galaxies, supporting the correlation between star formation density and starburst activity that is measured for the local sample. Locally, luminous and ultraluminous IR galaxies, (U)LIRGs (L tot IR ≥ 10 11 L ), are systematically in the starburst mode, whereas most distant (U)LIRGs form stars in the "normal" main sequence mode. This confusion between two modes of star formation is the cause of the so-called "mid-IR excess" population of galaxies found at z > 1.5 by previous studies. Main sequence galaxies have strong polycyclic aromatic hydrocarbon (PAH) emission line features, a broad far-IR bump resulting from a combination of dust temperatures (T dust ∼ 15-50 K), and an effective T dust ∼ 31 K, as derived from the peak wavelength of their infrared SED. Galaxies in the starburst regime instead exhibit weak PAH equivalent widths and a sharper far-IR bump with an effective T dust ∼ 40 K. Finally, we present evidence that the mid-to-far IR emission of X-ray active galactic nuclei (AGN) is predominantly produced by star formation and that candidate dusty AGNs with a power-law emission in the mid-IR systematically occur in compact, dusty starbursts. After correcting for the effect of starbursts on IR8, we identify new candidates for extremely obscured AGNs.