The success of nature-based tourism destinations depends on the sustainable use of common pool resources (CPRs). More often than not, tourism demands compete for these resources, exerting pressure on them, resulting in decline of the CPRs. Managing tourism and environmental resources has become extremely important, but also more complex, as the interests of different stakeholders are intertwined across international, national, and local levels. Hence, this study aimed to investigate how stakeholder groups perceive the issues relating to the environment using Yakushima Island, Japan for a case study. Quantitative data were collected by administering a questionnaire to residents and tourism practitioners, while qualitative data were gathered through semi-structured interviews conducted with organizations involved with tourism and environmental management in Yakushima. The study reveals that the underlying environmental issues in Yakushima result from an increase in tourists and controversial management of deer populations. Both residents and tourism practitioners indicated that more trash in the community was the most significant impact of tourism on the environment. Some perceptions were contradictory. Management authorities noted an increase in the deer population, which needs to be controlled to prevent damage to natural vegetation. Most tour guides reported a decrease in the sighting of deer along hiking trails, frustrating the genuine expectations of the tourist. The connection of the deer issue to tourism turned out to be very complex, as an overabundance of deer endangers parts of the vegetation tourists come to enjoy, while at the same time deer are an important part of the tourists’ experience. Different perceptions on deer by different stakeholders add to this complexity, which needs to be taken into consideration for proper management of tourists and deer in the future.