As the importance of intrusion detection and prevention systems (IDPSs) increases, great costs are incurred to manually manage the signatures that are generated by malicious communication pattern files. Experts in network security management need to classify signatures by importance for an IDPS to work optimally. In this study, we propose and evaluate a machine learning signature classification model with a reject option to reduce the cost of setting up an IDPS. To train the proposed model, it is essential to design features that are effective for signature classification. Experts classify signatures with predefined if-then rules. An if-then rule returns a label of low, medium, high, or unknown importance based on keyword matching of the elements in the signature. Therefore, we first design two types of features, symbolic features (SFs) and keyword features (KFs), which are used in keyword matching for the if-then rules. Next, we design web information and message features (WMFs) to capture the properties of signatures that do not match the if-then rules. The WMFs are extracted as term frequency-inverse document frequency (TF-IDF) features of the message text in the signatures. The features are obtained by web scraping from the referenced external attack identification systems described in the signature. Because failure needs to be minimized in the classification of IDPS signatures, as in the medical field, we consider introducing a reject option in our proposed model. The effectiveness of the proposed classification model is evaluated in experiments with two real datasets composed of signatures labeled by experts: (i) a dataset that can be classified with if-then rules and (ii) a dataset with elements that do not match an if-then rule. In the experiment, the proposed model is evaluated from two perspectives: classification accuracy and reject option performance. In both cases, the combined SFs and WMFs performed better than the combined SFs and KFs. We also show that using an ensemble of neural networks improves the performance of the reject option. An analysis shows that experts refer to natural-language elements in the signatures and information from external information systems on the web.