Exploration and remote sensing with mobile robots is a well known field of research, but current solutions cannot be directly applied for tethered robots. In some applications, tethers may be very important to provide power or allow communication with the robot. This paper presents an exploration algorithm that guarantees complete exploration of arbitrary environments within the length constraint of the tether, while keeping the tether tangle-free at all times. While we also propose a generalized algorithm that can be used with several exploration strategies, our implementation uses a modified frontier-based exploration approach, where the robot chooses its next goal in the frontier between explored and unexplored regions of the environment. The basic idea of the algorithm is to keep an estimate of the tether configuration, including length and homotopy, and decide the next robot path based on the difference between the current tether length and the shortest tether length at the next goal position. Our algorithm is provable correct and was tested and evaluated using both simulations and real-world experiments.