Purpose: To develop a pipeline that automatically classifies patients for pulmonary embolism (PE) in CT pulmonary angiography (CTPA) examinations with high sensitivity and specificity. Materials and Methods: Seven hundred non-ECG-gated CTPA examinations from 652 patients (median age 72 years, range 16-100 years; interquartile range 18 years; 353 women) performed at a single institution between 2014 and 2018, of which 149 examinations contained PE, were used for model development. The nnU-Net deep learning-based segmentation framework was trained and validated in 5-fold cross-validation. To enhance classification, we applied logical rules based on PE volume and probability thresholds. External model testing was then performed in 770 and 34 CTPAs from two independent datasets. Results: For patient-level classification, a threshold PE volume of 20 mm3resulted in the best balance between sensitivity and specificity. In internal cross-validation and test set, the trained model correctly classified 123 of 128 examinations as positive for PE (sensitivity 96%; 95% C.I. 91-98%) and 521 of 551 as negative (specificity 95%; 95% C.I. 92-96%). In the first external test dataset, the trained model correctly classified 31 of 32 examinations as positive (sensitivity 97%; 95% C.I. 84-99%) and 2 of 2 as negative (specificity 100%; 95% C.I. 34-100%). In the second external test dataset, the trained model correctly classified 379 of 385 examinations as positive (sensitivity 98%; 95% C.I. 97-99%) and 346 of 385 as negative (specificity 90%; 95% C.I. 86-93%). Conclusion: Beyond state-of-art classification for PE in CTPA was achieved using nnU-Net for deep learning-based segmentation in combination with volume- and probability-based classification.