We present an analysis of the deepest Herschel images in four major extragalactic fields GOODS-North, GOODS-South, UDS, and COSMOS obtained within the GOODS-Herschel and CANDELS-Herschel key programs. The star formation picture provided by a total of 10 497 individual far-infrared detections is supplemented by the stacking analysis of a mass complete sample of 62 361 starforming galaxies from the Hubble Space Telescope (HST) H band-selected catalogs of the CANDELS survey and from two deep ground-based K s band-selected catalogs in the GOODS-North and the COSMOS-wide field to obtain one of the most accurate and unbiased understanding to date of the stellar mass growth over the cosmic history. We show, for the first time, that stacking also provides a powerful tool to determine the dispersion of a physical correlation and describe our method called "scatter stacking", which may be easily generalized to other experiments. The combination of direct UV and far-infrared UV-reprocessed light provides a complete census on the star formation rates (SFRs), allowing us to demonstrate that galaxies at z = 4 to 0 of all stellar masses (M * ) follow a universal scaling law, the so-called main sequence of star-forming galaxies. We find a universal close-to-linear slope of the log 10 (SFR)-log 10 (M * ) relation, with evidence for a flattening of the main sequence at high masses (log 10 (M * /M ) > 10.5) that becomes less prominent with increasing redshift and almost vanishes by z 2. This flattening may be due to the parallel stellar growth of quiescent bulges in star-forming galaxies, which mostly happens over the same redshift range. Within the main sequence, we measure a nonvarying SFR dispersion of 0.3 dex: at a fixed redshift and stellar mass, about 68% of star-forming galaxies form stars at a universal rate within a factor 2. The specific SFR (sSFR = SFR/M * ) of star-forming galaxies is found to continuously increase from z = 0 to 4. Finally we discuss the implications of our findings on the cosmic SFR history and on the origin of present-day stars: more than two-thirds of present-day stars must have formed in a regime dominated by the "main sequence" mode. As a consequence we conclude that, although omnipresent in the distant Universe, galaxy mergers had little impact in shaping the global star formation history over the last 12.5 billion years.