Earlier this week, Telus released the results of their 2009 joint Telus/Rotman School of Management at the University of Toronto study on Canadian IT Security Practices, which claimed that the number of breaches tripled to an average of 11.3. Here is the press release. But are these valid claims?
First, let's take a deeper look at the average of 11.3. By simply taking the raw answers of the 2009 question about the number of breaches during the last 12 months, the average, i.e. mean, is indeed 11.3. However, let's take a closer look at the actual responses:
Number of Breaches Percentage of Organizations
0 14%
1 6%
2 to 5 33%
6 to 10 9%
11 to 25 7%
26 to 50 3%
51 to 100 2%
More than 100 2%
Don't know 23%
Given the number of outliers, the average (mean) is not really a valid number. Those outliers significantly skew the average. The mode, between 2 and 5, is much more meaningful.
Also, there is no attempt to correlate the number of breaches an organization suffered with the organization's size. Of the 500 organizations participating, 31% had under 100 employees and 23% had over 10,000 employees. The point here is that the outliers may very well be a small group of very large organizations.
Now let's address the claim of "tripling." What could account for this huge increase?
- It may just be a case of people being more honest this year, i.e. reporting more breaches. After all, this is just a survey.
- It may be that organizations actually have better security controls in place and therefore detected more breaches.
- It may be a function of the organizations participating. In 2008, there were only 297 versus 500 in 2009.
- It could be the change in the way the question was worded in 2009 versus 2008. Here is the question from 2008 (In fact the only place in the study that uses real numbers rather than percentages):
Q40. A major IT security incident can be defined as one which causes a disruption to normal activities such that significant time, resources, and/or payments are required to resolve the situation. Based on this definition, how many major security incidents do you estimate your organization has experienced in the last 12 months?
1 to 5 63%
6 to 10 2%
More than 10 1%
Don't know 24%
The 2009 study question:
Q48. How many Security breaches do you estimate your organization has experienced in the past 12 months?
I provided the responses earlier in this post. The point is that in 2008, the question specifically asked about major incidents while in 2009 the question was about all breaches.
Also note, in both cases the organizations were asked for "estimates." Don't most of these organizations have Security Incident Response Teams? At least the 69% with over 100 full time employees? Wouldn't they know exactly how many "incidents" they investigated and how many were actual breaches?
I suppose studies like these, based on surveys, have some value, but we really need information based on facts and analysis based on sound techniques.
