We argue that a unitary description of the formation and evaporation of a black hole implies that the Bekenstein-Hawking entropy is the "entropy of a vacuum": the logarithm of the number of possible independent ways in which quantum field theory on a fixed classical spacetime background can emerge in a full quantum theory of gravity. In many cases, the covariant entropy counts this entropy-the degeneracy of emergent quantum field theories in full quantum gravity-with the entropy of particle excitations in each quantum field theory giving only a tiny perturbation. In the Rindler description of a (black hole) horizon, the relevant vacuum degrees of freedom manifest themselves as an extra hidden quantum number carried by the states representing the second exterior region; this quantum number is invisible in the emergent quantum field theory. In a distant picture, these states arise as exponentially degenerate ground and excited states of the intrinsically quantum gravitational degrees of freedom on the stretched horizon. The formation and evaporation of a black hole involve processes in which the entropy of collapsing matter is transformed into that of a vacuum and then to that of final-state Hawking radiation. In the intermediate stage of this evolution, entanglement between the vacuum and (early) Hawking radiation develops, which is transferred to the entanglement among final-state Hawking quanta through the evaporation process. The horizon is kept smooth throughout the evolution; in particular, no firewall develops. Similar considerations also apply for cosmological horizons, for example for the horizon of a meta-stable de Sitter space.