Background Physical fitness has been shown to predict cardiovascular death during long-term follow-up. In the present study we aimed to investigate how physical fitness and other cardiovascular risk factors at middle-age influenced the risk of cardiovascular death during the early (0-11 years), intermediate (12-23 years) and late (24-35 years) parts of a 35-year observation period. Methods and results Age-adjusted physical fitness was calculated in 2014 apparently healthy, middle-aged men after maximal bicycle electrocardiogram-tests in 1972-1975 (Survey 1) and 1979-1982 (Survey 2). The men were assessed through 35 years after Survey 1, and 28 years after Survey 2 by Cox proportional hazards models. Low Survey 1 physical fitness was independently associated with increased risks of early and intermediate, but not late, cardiovascular death. Survey 1 to Survey 2 change in physical fitness, age, smoking status, systolic blood pressure and cholesterol impacted cardiovascular death risks in all periods. Family history of coronary heart disease impacted early and intermediate, but not late, cardiovascular death. Conclusions Most classical cardiovascular risk factors were strong predictors of early, intermediate and late cardiovascular death. Physical fitness measured at median age 50 years was independently associated with risk of early cardiovascular death, but the association weakened as time progressed. Change in physical fitness during middle-age impacted cardiovascular death risk in a full lifetime perspective. Thus, our data suggest that physical fitness is a modifiable cardiovascular risk factor with limited duration in contrast to the sustained impact of smoking, blood pressure and cholesterol on cardiovascular mortality.