Theory and observations agree that the accreted stellar halos (ASHs) of Milky Way-like galaxies display significant scatter. I take advantage of this stochasticity to invert the link between halo assembly history (HAH) and ASH, using mock ASHs corresponding to 750 ΛCDM HAHs, sharing a final virial mass of M h (z = 0) = 10 12.25 M . Hosts with poor/rich ASHs assemble following orthogonal growth-patterns. Hosts with rich ASHs experience accretion events (AEs) with high virial mass ratios (HVMRs, M s /M h 0.1) at 0.5 z inf all 1.5, in a phase of fast growth. This maximizes the accreted stellar mass under the condition these satellites are disrupted by z = 0. At similar times, hosts with poor ASHs grow slowly through minor mergers, with only very recent HVMR AEs: this results in a globally more abundant satellite population and in distinctive surviving massive satellites (stellar mass log M s, * /M 9). Several properties of the Milky Way are in agreement with the predictions of this framework for hosts with poor, concentrated ASHs, including: i) the recent infall of Sagittarius and Magellanic Clouds, ii) the likely higher-than-average concentration of its dark halo, iii) the signatures of fast chemical enrichment of a sizable fraction of its halo stellar populations.