We study the dependence of the embedding calculus Taylor tower on the smooth structures of the source and target. We prove that embedding calculus does not distinguish exotic smooth structures in dimension 4, implying a negative answer to a question of Viro. In contrast, we show that embedding calculus does distinguish certain exotic spheres in higher dimensions. As a technical tool of independent interest, we prove an isotopy extension theorem for the limit of the embedding calculus tower, which we use to investigate several further examples. 58D10; 55P48, 57N35, 57R40 1. Introduction 353 2. Preliminaries 355 3. Formally smooth manifolds 363 4. Embedding calculus in dimension 4 370 5. Embedding calculus and exotic spheres 374 6. Isotopy extension for embedding calculus 377 Appendix. Homotopy pullbacks of simplicial categories 386 References 390