An In-Depth Benchmarking and Evaluation of Phishing Detection Research for Security Needs

Abstract

We perform an in-depth, systematic benchmarking study and evaluation of phishing features on diverse and extensive datasets. We propose a new taxonomy of features based on the interpretation and purpose of each feature. Next, we propose a benchmarking framework called 'PhishBench,' which enables us to evaluate and compare the existing features for phishing detection systematically and thoroughly under identical experimental conditions, i.e., unified system specification, datasets, classifiers, and evaluation metrics. PhishBench is a first in the field of benchmarking phishing related research and incorporates thorough and systematic evaluation and feature comparison. We use PhishBench to test methods published in the phishing literature on new and diverse datasets to check their robustness and scalability. We study how dataset characteristics, e.g., varying legitimate to phishing ratios and increasing the size of imbalanced datasets, affect classification performance. Our results show that the imbalanced nature of phishing attacks affects the detection systems' performance and researchers should take this into account when proposing a new method. We also found that retraining alone is not enough to defeat new attacks. New features and techniques are required to stop attackers from fooling detection systems. INDEX TERMS Feature engineering, feature taxonomy, framework, phishing email, phishing URL, phishing website.

Perspectives on Adversarial Classification

“…In applications in which there is a computational bottleneck, the second approach may be preferable, with possible changes in the adversary's behaviour incorporated via retraining. This tension between the need to robustify algorithms against attacks (training phase, Section 5) and the fast adaptivity of attackers against defences (operational phase, Section 4) is well exemplified in the phishing detection domain as discussed in [2].…”

“…Classification is a major research area with important applications in security and cybersecurity, including fraud detection [1]: phishing detection [2], terrorism [3] or cargo screening [4]. An increasing number of processes are being automated through classification algorithms; it is essential that these are robust to trust key operations based on their output.…”

A comprehensive survey of AI-enabled phishing attacks detection techniques

“…Authors in El Aassal et al [ 22 ] proposed a benchmarking structure called PhishBench, which enables us to assess and analyze the existing features for phishing detection and completely understand indistinguishable test conditions, i.e., unified framework specification, datasets, classifiers, and performance measurements. The examinations indicated that the classification execution dropped when the proportion among phishing and authentic decreases towards 1 to 10.…”

Scam Augmentation and Customization: Identifying Vulnerable Users and Arming Defenders

“…Company representation fraud (aka job scam) attacks have increased in recent years [12,46,47]. Moreover, phishing detectors are not able to catch these frauds without retraining [1]. To gain an understanding of the parameters contributing to deception and action by the victim, we conduct a between-subjects study in which we take a well-known attack, viz., job scam, and parameterize it with signals such as surrounding context (facade) and customized content.…”