Vector problem-solving abilities are fundamental to everyday life and higher education; thus, improving them is important in education and research. However, the role of cognitive and affective factors and learning engagement in vector problem-solving performance is still unclear. This study examines the processes associated with vector problem-solving performance, focusing on the problem-solving strategy as a cognitive factor and math anxiety and task-specific self-efficacy as affective factors. In addition, this study examines the impact of learning engagement as a moderator in this process. A total of 245 Japanese 11th-grade high school students completed questionnaires. A multiple-group structural equation modelling revealed that (1) task-specific self-efficacy, math anxiety, and problem-solving strategies contribute to vector problem-solving performance when learning engagement is above average; (2) task-specific self-efficacy contributes to math anxiety, whereas task-specific self-efficacy and math anxiety contribute to problem-solving strategies when learning engagement is above average and stable; (3) task-specific self-efficacy is a positive predictor of vector problem-solving performance regardless of learning engagement. The results suggest that learning engagement moderates the association between math anxiety, task-specific self-efficacy, problem-solving strategy, and vector problem-solving performance. In addition, task-specific self-efficacy is a strong predictor of vector problem-solving performance.