Calibrating experts

All expert opinions are not equally useful. Some expert opinions aren't even very accurate. So in cases where you need some sort of expert advice, you need to be able to somehow calibrate the opinions of experts to make the opinions more useful. I do this when I decide which books to buy.

There are so many books in print that it's impossible to read a page or two from each of them to see if they're the sort of thing that appeals to you. Even in relatively small niches like genre fiction, there are so many books published each year that one person can't really keep up with all of them. Because of this, a useful alternative is to find someone whose job is to review books full-time and to use their opinions to help you make your decisions. There are also as many opinions as there are book reviewers, so you also need a way to filter the opinions of reviewers. What I've done in the past is to find people whose opinions tend to agree with mine and to then use their reviews to help me decide which books to get.

You have a similar problem with any specialized field, and information security is no exception to this. Fortunately, there are techniques that have been devised that let you calibrate the opinions of experts. These essentially generalize my technique for deciding which books to consider buying: you get expert opinions for which you know the right answers and use the accuracy of these answers to develop a way to interpret the opinions that you get when you don't know the right answer. If you formalize this technique using Bayesian statistics, you get the Mendel-Sheridan model that was first described in 1989 by Max Mendel and Thomas Sheridan in "Filtering Information from Human Experts," and you can probably use this approach to create a way to help interpret what consultants tell you.

If you're working with a consultant, it's likely that the consultant's knowledge will be better than yours in the area of his specialization. Because of this, you probably can't ask him meaningful questions about it. Using the Mendel-Sheridan approach, however, you'd ask questions that you do know the answers to and use the accuracy of his responses to form your opinion of the answers that you don't know. That's apparently the best way to handle the situation. You can probably even apply the same principle to finding the right consultant to begin with.

We probably already do this, of course, but it's sometimes comforting to know that there's a reasonable basis for how we do things.