Background Mucous membrane pemphigoid (MMP) is a rare autoimmune bullous disease predominantly affecting the oral mucosa. Optimal management relies upon thorough clinical assessment and documentation at each visit. Objectives The primary aim of this study was to validate the Oral Disease Severity Score (ODSS) for the assessment of oral involvement in MMP. We also compared its inter-and intraobserver reliability with those of the oral parts of the Mucous Membrane Pemphigoid Disease Area Index (MMPDAI), Autoimmune Bullous Skin Disorder Intensity Score (ABSIS) and Physician's Global Assessment (PGA). Methods Fifteen patients with mild-to-moderately severe oral MMP were scored for disease severity by 10 oral medicine clinicians from four U.K. centres using the ODSS, the oral sections of MMPDAI and ABSIS, and PGA. Two clinicians rescored all patients after 2 h. Results In terms of reliability, the interobserver ODSS total score intraclass correlation coefficient (ICC) was 0Á97, MMPDAI activity 0Á59 and damage 0Á15, ABSIS total 0Á84, and PGA 0Á72. The intraobserver ICCs (two observers) for ODSS total were 0Á97 and 0Á93; for MMPDAI activity 0Á93 and 0Á70 and damage 0Á93 and 0Á79; for ABSIS total 0Á99 and 0Á94; and for PGA 0Á92 and 0Á94. Convergent validity between ODSS and MMPDAI was good (correlation coefficient 0Á88). The mean AE SD time for completion of ODSS was 93 AE 31 s, with MMPDAI 102 AE 24 s and ABSIS involvement 71 AE 18 s. The PGA took < 5 s. Conclusions This study has validated the ODSS for the assessment of oral MMP. It has shown superior interobserver agreement over MMPDAI, ABSIS and PGA, and superior intraobserver reliability to MMPDAI. It is quick and easy to perform. What's already known about this topic? • There are no validated scoring methodologies for oral mucous membrane pemphigoid (MMP). • Proposed disease activity scoring tools for MMP include the Mucous Membrane Disease Area Index (MMPDAI) and the Autoimmune Bullous Skin Disorder Intensity Score (ABSIS). • The Oral Disease Severity Score (ODSS) has been validated for use in oral pemphigus vulgaris (PV). It has been shown to be reliable and sensitive in both lichen planus (LP) and MMP.