In this chapter the theoretical foundations for teaching the interplay of mathematics and physics are described from the stance of physics as a science. The role of mathematics in and for physics and vice versa has been exhaustively discussed mostly from the viewpoint of physicists and philosophers. However, in the light of the overwhelming significance of mathematics in its broadest sense for physics it is important to analyse which insights and knowledge about the role of mathematics for physics, students should acquire during their educational career. This is strongly connected to insight into the nature of physics. It will be elucidated, which different roles mathematics can take on in physics, how physicists use different mathematical elements, structures or algorithms, which of them are important as well for physics as a science as in physics education, how the meanings and conventions relate between mathematics and physics and which frameworks are used for understanding teaching and learning mathematics in physics. The relevance of mathematics for physics education and the epistemological aspects in the interplay are described giving valuable insight into the manner in which physicists shape and use the interplay and how the interplay shapes physics.