Introduction: The aim of this study was to investigate the association between physical fitness, physical activity, and sense of coherence (SOC) in Swedish adolescents (n = 2028 males, n = 1287 females) aged 14 to 18 using a cross-sectional design. Methods: Using the Swedish Physical power Mental harmony and Social capacity (FMS) student profile, participants performed physical tests measuring their cardiovascular ability (CV) and muscular strength. Questionnaires were used to measure physical activity levels (PA), the participation in organized physical training, sedentary behavior (SB), screen time, and SOC value. Multiple linear regression analyses were used to analyze the association between SOC and independent variables. Results: The regression analyses explained a small, shared variance between SOC and the independent variables in boys (4.3%) and girls (3.3%). SB showed a positive association with SOC both in girls (β = 0.114, p = 0.002) and boys (β = 0.109, p = 0.013). Abdominal strength was positively associated, while VO2max was inversely associated, with SOC (β = 0.113, p = 0.022; β = −0.109, p = 0.026, respectively) in boys. Girls had poorer fitness than boys did across all age groups except at age 18. PA levels decreased from age 14 to 18 in girls and boys, but without differences between sexes. Abdominal strength decreased from age 14 to 18 in girls and boys. In general, girls had lower SOC than boys across all age groups. Conclusions: Poor sedentary behavior was significantly associated with weaker SOC for both genders, indicating overall physical activity as the most important factor for stronger SOC. However, emotional support in vulnerable environments may have a bigger impact than physical activity or sedentary behavior on the SOC value for adolescents.