Phishing in Organizations: Findings from a Large-Scale and Long-Term Study

Abstract

In this paper, we present findings from a largescale and long-term phishing experiment that we conducted in collaboration with a partner company. Our experiment ran for 15 months during which time more than 14,000 study participants (employees of the company) received different simulated phishing emails in their normal working context. We also deployed a reporting button to the company's email client which allowed the participants to report suspicious emails they received. We measured click rates for phishing emails, dangerous actions such as submitting credentials, and reported suspicious emails.The results of our experiment provide three types of contributions. First, some of our findings support previous literature with improved ecological validity. One example of such results is good effectiveness of warnings on emails. Second, some of our results contradict prior literature and common industry practices. Surprisingly, we find that embedded training during simulated phishing exercises, as commonly deployed in the industry today, does not make employees more resilient to phishing, but instead it can have unexpected side effects that can make employees even more susceptible to phishing. And third, we report new findings. In particular, we are the first to demonstrate that using the employees as a collective phishing detection mechanism is practical in large organizations. Our results show that such crowd-sourcing allows fast detection of new phishing campaigns, the operational load for the organization is acceptable, and the employees remain active over long periods of time.

Avoiding the Hook: Influential Factors of Phishing Awareness Training on Click-Rates and a Data-Driven Approach to Predict Email Difficulty Perception

“…Generally speaking, phishing awareness training continually exposes the participants to the risk of wasting their time [87]. However, as Lain et al [74] stated in their study, it does not expose the participants to a greater risk than what they would encounter during their daily lives because the partici-pants are regularly exposed to real phishing emails or spam. We acknowledge that conducting this study exposed our participants to minimal risks but similar to other researchers [74], we believe that the positive experience the participants gain merited these risks.…”

“…However, as Lain et al [74] stated in their study, it does not expose the participants to a greater risk than what they would encounter during their daily lives because the partici-pants are regularly exposed to real phishing emails or spam. We acknowledge that conducting this study exposed our participants to minimal risks but similar to other researchers [74], we believe that the positive experience the participants gain merited these risks. In addition, the decision to conduct this study with all students and staff members was approved by the highest panel of our university (including the CISO) and we followed the ethical guidelines of our university to the best of our knowledge.…”

“…The literature on the efficiency of anti-phishing training is more consistent even when still many contradictions exist. For example, a recent study by Lain et al [74] concluded that conducting embedded training may not provide the wished training effects or can even have negative side effects. Other studies have focused on when to reapply the training [35].…”

“…One of the tough challenges for all researchers in this domain is that collecting the necessary data for such a study involves conducting anti-phishing training with several hundred or even thousands of participants, which holds some ethical-, legal-and organizational challenges. Nevertheless comparable studies have been conducted which show promising results in revealing the nature of anti-phishing training effects [19], [42], [45], [74]- [78]. However, to our knowledge, no other studies have been attempting to estimate the difficulty of phishing awareness training at this scale; Other papers had fewer participants and fewer training iterations [57]- [60], or did not focus on the estimation of difficulty [74], [76].…”

“…Nevertheless comparable studies have been conducted which show promising results in revealing the nature of anti-phishing training effects [19], [42], [45], [74]- [78]. However, to our knowledge, no other studies have been attempting to estimate the difficulty of phishing awareness training at this scale; Other papers had fewer participants and fewer training iterations [57]- [60], or did not focus on the estimation of difficulty [74], [76].…”