Why Standard Security Awareness Tool KPIs Don't Fully Represent Your Security Culture

Every organization wants a strong security culture! To achieve this, employees must be regularly trained and made aware of information security. It is important to set measurable goals - and this requires key performance indicators. Most vendors provide such metrics in their training and phishing tools, which give a good overview of the training that has been done. Unfortunately, this is not enough. To truly strengthen your security culture, you need more: With our Security Awareness Radar®, we have a solution for you.

There is nothing more practical than turning to an external provider to strengthen your security culture. They provide all the training materials such as e-learning, explanatory videos, Phishing simulations, etc., and you take care of your day-to-day business. As a CISO or security officer, you can report on the metrics provided to provide management with an overview of the results of the training conducted.

What many people do not realize is that these metrics tell little about the actual security culture in the organization. But what metrics are needed to learn more about the factors that influence security behavior? Why should you know these metrics? Isn't it enough to use the metrics from the training tools? Why is it useful to identify additional metrics? What's the best way to do it?

In this blog post, we will provide sound answers to all of these questions.

What metrics do training tools collect?

Depending on the vendor or training tool, the metrics vary. However, most collect the following data:

For e-learning and campaigns

• Topic of the e-learning or campaign.
• Number of employees invited.
• Number of courses completed.
• Number of active courses.
• Number of courses cancelled.
• Number of courses not started.
• Campaign or e-learning start and end dates.
• Average time to complete a course.
• Area-specific reports.
• Average time between receipt of invitation email and start of e-learning course.

For phishing simulations

• Number of phishing simulation emails delivered.
• Number of attachments opened in phishing simulation emails.
• Number of links clicked in phishing simulation emails.
• Number of reported simulated phishing emails.
• Number of undelivered simulated phishing emails.
• Number of unopened simulated phishing emails.
• Response time after receiving phishing simulation emails.
• Date and time phishing simulation emails were opened.
• Which department completed and how?

What do these metrics tell us?

All of the metrics listed can be used to meet compliance requirements, prepare for an audit, or support certification. They are also often used to provide feedback and to justify further action to management. They show what has been trained, how employees have performed, and how the success rate is evolving.

The metrics also tell you how many employees were reached by the campaigns and how many of them successfully completed the training. In addition, you will receive the following information:

1. Click-through rate on the phishing simulation email, link, or attachment: The click-through rate on a link, for example, indicates how many employees clicked on a potentially malicious link. The higher the percentage of employees who clicked on the link, the greater the need for training. The same applies to clicks on the attachment or opening the phishing simulation email.
2. Recognition rate of the phishing simulation email: The recognition rate indicates how many employees are able to recognize a phishing email and flag it as such. A higher recognition rate indicates that employees are better prepared to deal with potential threats.
3. Time to report: How quickly employees report a suspicious email after it is detected also indicates the level of employee knowledge. Quick reporting indicates that employees are better equipped to respond to potential threats.
4. Training participation: Training participation indicates how many employees attended the training. A high participation rate may indicate that the training was relevant and useful to employees.
5. Error rate: The error rate indicates how often employees failed to recognize phishing simulation emails. A low error rate indicates that the training was successful and that employees are better able to recognize potential threats. However, it can also mean that the phishing email simulations were too easy.
6. Employee feedback: Employee feedback indicates how useful and relevant they found the training. Positive feedback means that the training met employees' needs and expectations.

Looking at other metrics, such as the number of reported security incidents or actual phishing emails, can help determine whether the training has improved day-to-day operations. The more employees who have successfully completed the training (e-learning, phishing simulations, campaigns), the more accurate the trend will be as to whether the training has improved security awareness.

However, it is important to note that all metrics are snapshots. They reflect the knowledge of employees at a particular point in time. Therefore, they should not be viewed in isolation, but in conjunction with other relevant KPIs.

How useful are the metrics in terms of further training?

The metrics can be used to determine which topics need more training. It can also be used to determine if there is an increased need for training on a particular topic in certain departments or regions.

The metrics collected can be used to better evaluate the success of current training. The data can also be used to identify potential weaknesses. These can then be improved accordingly.

Who are the metrics for?

The metrics from the training tools are of particular interest to CISOs and security managers, as they can use them to report to management and the board. Management and the board are usually interested in up-to-the-minute numbers that show the effectiveness of an action immediately after it is taken. This is often financially driven, as successful numbers make it more likely that more funding will be approved for more training.

Do the metrics from the training tools say anything about the security culture in the organization?

This is actually a weakness of the metrics from the training tools, because a security culture refers to the attitudes, beliefs, and behaviors of employees and managers regarding information security in the organization. While the metrics can give an indication of the security culture in the organization, they are only a snapshot because they were collected immediately after a training session. As such, they provide a good indication of the training itself, but say too little about how the security culture is practiced in the organization. In the case of ongoing training, such as phishing simulations, this is only true to a limited extent, as these allow more specific statements to be made about employee behavior.

Training tools alone cannot provide more specific information about the behavior and culture in which information security is practiced. For example, the KPIs from a training tool after "Clear Desk and Clear Screen" training do not show whether the instructions are actively implemented, screens are locked, and workplaces are left tidy.

When employees are well trained and aware of the importance of information security, they are more likely to be alert to threats and follow security protocols. However, if employees frequently ignore or fail to respond appropriately to security protocols, this may indicate a weakness in the organization's security culture.

Therefore, it is useful to collect additional metrics to improve security culture over time.

What other metrics are there to measure an organization's security culture?

The questions in security culture surveys, such as our Security Awareness Radar® survey, are much more aimed at employee behavior. How they behave, but also what behavior they observe in their colleagues, such as whether or not they lock their PCs when they leave the workplace.

The training topics are reflected in the security culture survey. However, the survey goes much deeper into employee behavior.

To truly understand and improve a security culture, questions should be asked not only about the training itself, but also, for example, about how the organizational culture and structure are perceived, or how communication and role modeling are practiced within the organization: How are problems handled? What are the values and attitudes of employees toward information security?

‍

Other benefits of measuring your security culture

• Benchmarking: External benchmarking, or comparison with other companies in the same industry and of a similar size, shows how the company stacks up against its competitors. Internal benchmarking allows for comparisons across different divisions or regions, to better identify which topics need more training and where.
• Strengthen the role model function: If, for example, a security culture survey shows that management and executives are not setting the right example for security, the CISO knows that this group needs more training. This, in turn, promotes secure behavior among employees. After all, why should I lock my PC if my boss doesn't?
• Feedback is important: If these surveys are also quantitative and qualitative in nature, i.e. if participants have the opportunity to provide feedback via a comment field, even more interesting information can be gathered. For example, the comments may indicate that employees generally want more security training.

So does it make sense to measure security culture as well?

Yes, absolutely! The metrics from the training tools are good and provide important information about the training itself. But they tell us too little about whether and how a security culture is lived in the company.

A lived security culture significantly increases the security emanating from the "human firewall". As a CISO or security manager, this gives you even more insight into who, what and how needs to be trained. What topics are relevant? Which departments or regions need more training attention?

In any case, it is important to train your employees regularly, across multiple channels, with engaging and valuable content. This will significantly improve the security culture and behavior of your employees, helping to strengthen the "human firewall".

Measuring security culture with the Security Awareness Radar®

Our Security Awareness Radar® not only shows which factors have a significant influence on security behavior and where possible weaknesses in information security and security culture exist in the organization, but also provides concrete suggestions for action to improve them. Among other things, these suggested measures also indicate whether and how a topic contributes to ISO certification.

‍

If a measurement is carried out regularly, e.g. every 2 years, the change and development of the security culture becomes visible. This helps to demonstrate the long-term success of training to management.

Want to learn more about how the Security Awareness Radar® (SAR®) works? Read our blog post.

Conclusion

Training providers' metrics give you a good overview of the success of the training conducted. However, they provide too little information about whether and how a security culture is lived in the company. To find out, a more in-depth measurement is required. Our Security Awareness Radar® is a useful analysis and planning tool. It supports you and your organization in strengthening your security culture.

Would you like to have your security culture measured? Then don't hesitate to schedule an appointment with one of our consultants today. We look forward to working with you to improve your security culture.

‍

‍

Subscribe to our newsletter now and never miss more information security and security awareness news and blogs. Subscribe using the form below.

‍