Trust, but verify

At the IEEE Symposium on Massive Storage Systems and Technologies last month, there were a number of vendors with booths in the exhibit area. Almost all of these were storage vendors. It was interesting hearing their pitches. They were quite different from the ones that you hear at information security shows. Here's the generic information security pitch that's commonly used today:

More than <random big number> data breaches occur each <random unit of time> causing over <random big number> dollars of loss annually

Source: <reference to an analyst report that's too vague to actually verify>

And the only way to stop this massive problem is to buy my product

The pitches of storage vendors seem to be different, yet also similar in some ways to the pitch that security vendors use. Here's a generic version of what I frequently heard at MSST 2008:

The amount of data that your business handles is growing at a rate of over <random big number> of terabytes per <random unit of time>

Source: <vague reference to some obscure academic paper>

And the only way to deal with this huge amount of data is to buy my product

One common fact that was mentioned was that it's been estimated that the amount of information will be doubling every 11 hours by the year 2010.

Yikes! That’s fairly soon. It looks like I'd better buy some of the leading-edge technologies that I saw at MSST 2008.

Wait a minute. The year 2010 isn't really that far off, and I don't see information doubling at anything approaching that rate. Because of this, I looked up the reference that the storage vendors point to. Here's the reference:

Nick Bontis and Jac Fitz-enz, "Intellectual capital ROI: a causal map of human capital antecedents and consequents," Journal of Intellectual Capital, Vol. 3, No. 3, 2002, pp. 223-247.

It turns out that this paper, in turn, just refers to an earlier one:

Nick Bontis, "Managing organizational knowledge by diagnosing intellectual capital: framing and advancing the state of the field," International Journal of Technology Management, Vol. 18, Nos. 5/6/7/8, 1999, pp. 433-463.

When I finally found a copy of this paper, here's what it said:

"The more knowledge is supplied (or shared) the more highly it is valued. Furthermore, when was the last time the demand for knowledge went down? In fact, scientific folklore in the early 1900s stated that all the information in the world doubled every 30 years. As the 1970s approached, that number was reduced to seven years. Prognosticators have pushed this notion further and state that by the year 2010 all the information in the world will double every 11 hours."

You can't get much vaguer than "prognosticators have pushed this notion further" without any further reference cited, can you? And when the paper containing that vague claim was then cited by other papers, there was no obvious way to see that it was probably not to be taken seriously. That took a bit of detective work.

So it certainly seems that there's probably no basis for believing that the information in the world will really be doubling every 11 hours in just a couple of years. That doesn't mean that you won't need lots of storage then. It just won't be so much storage that you'll need to learn what zetabytes and yottabytes are any time soon.