Although the importance of virus-specific cytotoxic T lymphocytes (CTL) in virus clearance is evident in COVID-19, the characteristics of virus-specific CTLs related to disease severity have not been fully explored. Here we show that the phenotype of virus-specific CTLs against immunoprevalent epitopes in COVID-19 convalescents might differ according to the course of the disease. We establish a cellular screening method that uses artificial antigen presenting cells, expressing HLA-A*24:02, the costimulatory molecule 4-1BBL, SARS-CoV-2 structural proteins S, M, and N and non-structural proteins ORF3a and nsp6/ORF1a. The screen implicates SARS-CoV-2 M protein as a frequent target of IFNγ secreting CD8+ T cells, and identifies M198–206 as an immunoprevalent epitope in our cohort of HLA-A*24:02 positive convalescent COVID-19 patients recovering from mild, moderate and severe disease. Further exploration of M198–206-specific CD8+ T cells with single cell RNA sequencing reveals public TCRs in virus-specific CD8+ T cells, and shows an exhausted phenotype with less differentiated status in cells from the severe group compared to cells from the moderate group. In summary, this study describes a method to identify T cell epitopes, indicate that dysfunction of virus-specific CTLs might be an important determinant of clinical outcomes.