Which are the limits of a SDVA?

Written by Enrico Frumento, CEFRIEL

Today most of the companies are increasingly doing simulated phishing campaigns to test or train the vulnerability of their human layer of security, in other words, how easily their employees are falling into the threat of phishing. When the DOGANA project started, two years ago, the market was almost inexistent. Today instead, we assist to a growing number of companies offering simulated phishing frameworks, not considering the several companies that are doing simulated phishing tests on their own, without the help of any framework.

The market of simulated phishing and efficient training for the mitigation of the human layer of security is rapidly growing in the US, because of several big acquisitions. The followings are the most important for the autumn 2017:

  • Barracuda acquiring PhishLine[1],
  • Wombat Security acquired by Proofpoint for $225 million in cash[2],
  • Rapid7’s addition of a new simulated phishing testing product[3],
  • security awareness training and simulated phishing firm KnowBe4 secured $30 million in Series B financing, which brought the total amount raised by KnowBe4 to $44 million[4],
  • PhishMe was acquired for a $400 million deal to launch the new brand for training services Cofense[5]

The combination of simulated phishing (for risks assessment) and advanced/immersive training is one of the hottest sectors in security mitigation measures in today’s ICT market and one of those that moves the biggest source of revenues (mainly in American market[6], as demonstrated by the American Knowbe4[7] which announced a grew of 191% in the Q1 2018 ).

However, in the European context, a simulated phishing vulnerability assessment[8] and a simulated immersive training solutions[9] would be complex operations from the GDPR point of view.


How SDVAs cover the human layer threat landscape

The general aim of the DOGANA methodology is to be realistic as much as possible, simulating what an attacker would do without actually compromise the enterprise and committing illicit actions (as for any assessment), and it goes usually thorough these steps:

  1. Setup: support the company in internal sharing, especially with the involved stakeholder from different departments (HR, communication, labour parties, etc.).
  2. Passive social information mining and OSINT: perform a social network scanning and analysis using our own selected toolset, aimed to retrieve information or risk profile related to the company or its employees that could potentially be used by an attacker[10].
  3. Spear phishing attack simulation: perform a spear phishing campaign against a sample of the company employees’ population, selected in collaboration with the HR departments that potentially matches the profiles we identified on the previous phase.
  4. Technological attack simulation: perform a controlled technological attack aimed to assess the overall level of security of the enterprise against these specially crafted APT-like attacks, a step that is usually performed through ad-hoc malware, not able to infect the systems, but properly created around the victim’s characteristics.
  5. Awareness: setup and develop custom innovative awareness experiences with the purpose to train employees toward the adoption of longer lasting and safer habits.

However, there is a crucial question not still answered: which is the distance among an SDVA and real attacks? In other words, which is the capability of an SDVA to simulate real phishing attacks? This is a crucial question because SDVAs, like any other assessment methodology, are useful only if they are able to “cover” the threat landscape of an enterprise and hence, measure the corresponding cyber risks.


Are SDVAs able to emulate highly contextualized phishing attacks and are they useful for an enterprise?

Figure 1 shows (see Frumento, Puricelli, DeepSec 2014[11]) where the phishing tests we usually run in an SDVA are placed in a Contextualization-Volumes space.

The Volumes axis refers to the number of identical email sent in a phishing campaign and has three values:
(1) mails sent to few selected victims;
(2) mail sent to a subset of the whole company (e.g., a department);
(3) mail spread to all the employees.

Figure 1 – A conceptualization of the mail space useful to identify the type of phishing tests performed during an SDVA


The Contextualization axis refers to the degree of contextualization the email has (e.g., custom graphic, sensible topics, personal argumentations) and has three values:
(1) generic, thus not customized at all;
(2) company, thus properly customized for the specific enterprise (e.g., use of the official look or logo);
(3) person, thus contextualized to a single person’s interests.

To help understanding this classification we placed in this space the classic and the RSA phishing samples[12],[13]  as references. This graphical taxonomy helps to spot four important areas:

  • Today unfeasible attacks: phishing customized at personal level, but spread to a large number of victims. This area will become popular with the improvement of the semantic and sentiment analysis technologies.
  • Anti-economic attacks: phishing attacks targeted at few selected people, but not customized at all, are not economically sustainable nor convenient.
  • The upper left corner, where is the RSA sample, is unfeasible in the SDVAs due to legal reasons (i.e. this test would require active SM scanning).
  • The lower right corner, where is the classic phishing sample, is not useful to be tested with a SDVAs (i.e. companies already have lot of samples of this type).

Hence, the most convenient place for the SDVAs email tests is the grey area in the center. The dotted circle reports a legally possible, but nowadays still unexplored, extension of SDVAs on restricted groups for very mission critical employees (e.g., only directors or restricted project teams).

What comes out is a schematic representation of capabilities of an SDVA to cover the threat landscape of phishing. Hence, an SDVA has several degrees of approximation compared to a real social engineering attack: technical, ethical/legal and psychological.

  • Technical limitations are common to any vulnerability assessment methodology: the capability of a vulnerability assessment to emulate real attacks follows the limits of the instruments and the knowledge of the penetration testers. The vulnerability assessments for the technological layers have similar biases: first of all the capability to emulate the cyber-attack’s technological infrastructure.
  • Legal and ethical limits are an obvious consequence of the EU legislative framework (e.g. GDPR),
  • Psychological limits are a consequence of the still incomplete understanding of the human behavioural patterns in the cyber space and because there is a difference between a real unaware victim and an employee: as an example the victim may recognize that the mail is a phishing sample test and not a real one and behave differently.

The usefulness of an SDVA is directly proportional to the ability to emulate real attacks and assess the corresponding vulnerability of enterprises. Hence, these limitations are biases, which alter the effectiveness of the vulnerability assessment. In other words, throwing phishing email just to see if the employees click is almost useless. The phishing email must be approximately similar to those used by the attackers or, better, able to anticipate trends.


Aggressiveness of phishing emails

With reference to Figure 2, the aggressiveness of a phishing email is the level of customization of the phishing attack: the aggressiveness is directly proportional to the Contextualization and partially to the Volumes. At high Contextualization levels the Volumes becomes a relevant factor of complexity while it is irrelevant if the contextualization level is low (i.e., sending one or thousands generic –no contextualized- emails has the same difficulty nowadays).

Starting from this definition, we can state that the today’s aggressiveness of the real-world phishing emails is higher than the aggressiveness of the emails that are legally usable for the SDVAs.


Figure 2 – The aggressiveness of a phishing email is the level of customization of the phishing attack: the aggressiveness is directly proportional to the Contextualization and partially to the Volumes. At high contextualization levels the Volumes becomes a relevant factor of complexity.


The formula representing the aggressiveness is therefore the formula of a generic Cartesian plane passing through 3 points and the origin {0, 0, 0}.

This has some implications.

The first one is that the only reason why SDVAs work so well today is that thesecurity problems” of the enterprises’ human layer of security are usually big enough to be exploited by an SDVA email. In other words, the aggressiveness of an SDVA is high enough to exploit a company. In ICT security, this situation is mentioned as “bringing the low hanging fruits”. This situation is doomed to change as soon as the companies will improve the cyber posture of their human layer.

The second consideration is that the increased aggressiveness of today’s phishing attacks is a fundamental ingredient. Luckily, or unfortunately by the ICT security point of view, we cannot be that aggressive when doing an SDVA. This means that there is a distance among the SDVA tests and the real attacks, a grey area where we are not able to measure the vulnerability of the enterprise and therefore, not able to mitigate the corresponding risks. A concrete example is the following paradoxical situation: a company after repeated SDVAs and training campaigns starts to observe less positive results (i.e., the number of people who clicks in the phishing email decreases). Consequently, there is a perception of a safer cyber posture. Let suppose that a highly aggressive phishing breaks the systems few days after. Why this happened? It is because of the SDVA “limitations” or the because of the low preparation of the penetration testers?

One solution nowadays commonly used by many CISOs, is to use a moral hazard approach: do partially or completely illegally tests, doing more aggressive phishing, to obtain better cyber posture of the company. A practice that has its own risks and legal consequences.


[1] "Barracuda Acquires PhishLine", PR News wire, 2018. [Online]. Available: https://www.prnewswire.com/news-releases/barracuda-acquires-phishline-300576856.html.  

[2] "Proofpoint Enters into Definitive Agreement to Acquire Wombat Security Technologies for $225 million in Cash; Moves into Phishing Simulation and Security Awareness Training Market", GlobeNewswire News Room, 2018. [Online]. Available: https://goo.gl/89EWNp

[3] "Rapid7 Previews InsightPhish for Phishing Email Security", eWEEK, 2018. [Online]. Available: http://www.eweek.com/security/rapid7-previews-insightphish-for-phishing-email-security

[4] S. Sjouwerman, "Microsoft Confirms: "Sending Simulated Phishing Attacks to Your Employees Is a Must"", Blog.knowbe4.com, 2018. [Online]. Available: https://blog.knowbe4.com/microsoft-confirms-sending-simulated-phishing-attacks-to-your-employees-is-a-must.

[5] C. Osborne, "PhishMe acquired by private equity troupe, rebrands as Cofense", ZDNet, 2018. [Online]. Available: https://www.zdnet.com/article/phishme-acquired-by-private-equity-troupe-rebrands-as-cofense/

[6] DOGANA Consortium, “D 3.1 Report on existing tools their evaluation and the gap to be filled by DOGANA development”, 2018. [Onlie]. Available: https://www.dogana-project.eu/images/PDF_Files/D3.1-Report-on-existing-tools-their-evaluation-and-the-gap-to-be-filled-by-DOGANA-development.pdf

[7] S. Sjouwerman, "KnowBe4’s Year-Over-Year Sales Rocket 191% for Q1 2018", Knowbe4, 2018. [Online]. Available: https://blog.knowbe4.com/knowbe4s-year-over-year-sales-rocket-191-for-q1-2018

[8] DOGANA call these tests Social Driven Vulnerability Assessment (SDVA). An SDVA is a vulnerability assessment that concentrates on the human layer of security.

[9] See for example the “immersive” training solutions offered by Popcorn Training, which has been recently acquired by Knowbe4 https://www.knowbe4.com/press/knowbe4-expands-into-south-africa-by-acquiring-popcorn-training

[10] Active social network scanning is an alternative approach that usually involves the creation of a fake profile to help the attacker contacting the victim, but despite being a common attack technique, it is hardly feasible during an SVA, because of the several legal complications. Luckily, this is now not really an issue, because even without using active scanning the threat is quite relevant.

[11] “An innovative and comprehensive framework for Social Vulnerability Assessment”, E.Frumento, R.Puricelli, Magdeburger Journal zur Sicherheitsforschung, 8. Ausgabe, 4. Jahrgang, Band 2, 2014, available at: http://www.sicherheitsforschung-magdeburg.de/uploads/journal/MJS_033_Frumento_Assessment.pdf

[12] ‘Anatomy of an Attack’, RSA Blog, 01-Apr-2011. [Online]. Available: https://www.rsa.com/rivner/anatomy-of-an-attack.

[13] ‘Drive-by infections’, eBanking but secure. [Online]. Available: https://www.ebankingabersicher.ch/en/component/content/article/8-ihr-sicherheits-beitrag/erweiterter-schutz/75-drive-by-infektion.


by Enrico Frumento (CEFRIEL)


This project has received funding from the European Union’s Horizon 2020 Research and Innovation programme, under grant agreement No. 653618