Thoughts on penetration testing

In the recent X9F4 meeting we discussed the most recent draft of the X9.111 standard, Penetration Testing within the Financial Service Industry. There was lots of discussion of how disciplined, methodical and careful the penetration testing industry is these days. Most people seemed to think that this was a good thing. Some people even think that it’s a good idea to require your penetration testers to give you a complete list of all the tools that they plan to use in their testing, including the exact versions of the tools that they’ll be using.

On the other hand, I have to wonder if that sort of approach really gives an accurate impression of how well protected you are against hackers. After all, some hackers may use a similarly disciplined, methodical and careful approach, but others probably don’t. And if you restrict your testing to only attacks that the more disciplined hackers would use, you’ll almost certainly miss some of the attacks that less careful ones would use.

I’m out of touch with exactly how hackers operate these days, but I’d guess that most of them aren’t as careful and disciplined as professional penetration testers. If that’s the case, professional penetration tests may not really be giving you a good idea of how well you’re defended against hackers. But because that may not actually be the point of penetration tests, this may not really be a problem.