The recent discovery of high redshift dusty galaxies implies a rapid dust enrichment of their interstellar medium (ISM). To interpret these observations, we run a cosmological simulation in a 30h −1 cMpc/size volume down to z ≈ 4. We use the hydrodynamical code dustyGadget, which accounts for the production of dust by stellar populations and its evolution in the ISM. We find that the cosmic dust density parameter (Ω d ) is mainly driven by stellar dust at z 10, so that mass-and metallicity-dependent yields are required to assess the dust content in the first galaxies. At z 9 the growth of grains in the ISM of evolved systems (Log(M ⋆ /M ⊙ ) > 8.5) significantly increases their dust mass, in agreement with observations in the redshift range 4 z < 8. Our simulation shows that the variety of high redshift galaxies observed with ALMA can naturally be accounted for by modeling the grain-growth timescale as a function of the physical conditions in the gas cold phase. In addition, the trends of dust-to-metal (DTM) and dust-to-gas (D) ratios are compatible with the available data. A qualitative investigation of the inhomogeneous dust distribution in a representative massive halo at z ≈ 4 shows that dust is found from the central galaxy up to the closest satellites along polluted filaments with Log(D) −2.4, but sharply declines at distances d 30 kpc along many lines of sight, where Log(D) −4.0.