Add some rigor to information security

If you read scientific papers, you’ll find that there's a fair amount of precision in the way that data is analyzed, and that there’s a generally-accepted methodology for doing the analysis. This level of rigor is almost completely absent from studies about both the threats that information security deals with and the effectiveness of security technologies that address these threats. This makes making good decisions about information security harder than it needs to be.

Consider the fact that most people have an above-average number of legs. This is true, isn’t it? Most people have two legs, but a small number have either one or zero legs, so the average number of legs per person is probably close to 1.999. This means that most people actually do have an above-average number of legs. Without additional information about the distribution of the number of legs, it’s easy to jump to incorrect conclusions or to give unnecessary weight to this statement.

Similarly, it’s easy to create meaningless statistics about security. Suppose that most businesses have only one unit of security, but a few lucky ones have two units of security, making the average level of security close to 1.001. In this case, we can say that most businesses have a below-average level of security, can’t we? We can then act shocked that so many businesses don’t take security seriously. We might even try to lobby the government to create regulations that require a higher level of security as a way to address this problem. Without a more careful analysis of the data, it’s possible to reach conclusions that don’t make much sense.

So the next time a security vendor or industry analyst quotes statistics about threats or the effectiveness of security technologies, be sure to ask them for some additional detail to make sure that they are really telling you the full story. If they tell you what an average value is, ask them what the variance or standard deviation of the value is. If they show a model that predicts something, ask them about the sensitivity analysis that they did for the model. If they can’t answer questions like those, the numbers that they’re quoting may not be very useful.

There’s not much information available about threats or the effectiveness of security technologies, but there’s no reason to accept careless or sloppy analysis of the data that is available.