Chromosome conformation capture techniques (e.g Hi-C) reveal that intermediate-scale chromatin organization is comprised of “topologically associating domains” (TADs) on the tens to thousands of kb scale.1–5 The loop extrusion factor (LEF) model6–10 provides a framework for how TADs arise: cohesin or condensin extrude DNA loops, until they encounter boundary elements. Despite recent in vitro studies demonstrating that cohesin and condensin can drive loop formation on (largely) naked DNA11–13, evidence supporting the LEF model in living cells is lacking. Here, we combine experimental measurements of chromatin dynamics in vivo with simulations to further develop the LEF model. We show that the activity of the INO80 nucleosome remodeler enhances chromatin mobility, while cohesin and condensin restrain chromatin mobility. Motivated by these findings and the observations that cohesin is loaded preferentially at nucleosome-depleted transcriptional start sites14–18 and its efficient translocation requires nucleosome remodeling19–23 we propose a new LEF model in which LEF loading and loop extrusion direction depend on the underlying architecture of transcriptional units. Using solely genome annotation without imposing boundary elements, the model predicts TADs that reproduce experimental Hi-C data, including boundaries that are CTCF-poor. Furthermore, polymer simulations based on the model show that LEF-catalyzed loops reduce chromatin mobility, consistent with our experimental measurements. Overall, this work reveals new tenets for the origins of TADs in eukaryotes, driven by transcription-coupled nucleosome remodeling.