SARS-CoV-2 infection results in highly heterogeneous outcomes, from cure without symptoms to acute respiratory distress and death. While immunological correlates of disease severity have been identified, how they act together to determine the outcomes is unknown. Here, using a new mathematical model of within-host SARS-CoV-2 infection, we analyze diverse clinical datasets and predict that a subtle interplay between innate and CD8 T-cell responses underlies disease heterogeneity. Our model considers essential features of these immune arms and immunopathology from cytokines and effector cells. Model predictions provided excellent fits to patient data and, by varying the strength and timing of the immune arms, quantitatively recapitulated viral load changes in mild, moderate, and severe disease, and death. Additionally, they explained several confounding observations, including viral recrudescence after symptom loss, prolonged viral positivity before cure, and mortality despite declining viral loads. Together, a robust conceptual understanding of COVID-19 outcomes emerges, bearing implications for interventions.