Producing phosphorus (P) fertilizers with recycled P is desirable for efficient use of P resource. However, the current cost of P recycling facilities in Japan strongly discourages the government from adopting this practice. To expand consideration for a P recycling policy, the concept of social externality was introduced. Social issues, such as the violation of human rights in P mining in the Western Sahara, have been identified in recent studies; nevertheless, a systematic approach towards accountability was lacking. Therefore, the purpose of this study was to support a P recycling policy with a social life cycle assessment (SLCA) by contrasting the social impacts associated with mineral and recycled P fertilizers using the case study of Japan. We developed a framework based on the UNEP-SETAC SLCA Guidelines with a supplementary set of P-specific social indicators. The results showed that the marginal social impact associated with recycled P was much less relative to mineral P; however, even if we factored in the maximum recycling capacity, a mandate of P recycling policy in Japan would not mitigate the impacts significantly relative to the current situation because only 15% of P rocks could be substituted. In short, we showed that a semi-quantitative SLCA framework would be useful to communicate the wide spectrum of social impacts to policymakers.