Security-adjusted ROI

At the recent National Cyber Leap Year Summit, one of the ideas that was considered was research into developing a new metric called "security-adjusted ROI." I thought that this wasn't a good idea, and here's why.

ROI is the most popular metric used to justify security purchases, edging out NPV and IRR, the next most popular metrics by a comfortable margin. If you look at your favorite accounting text, you'll find careful definitions for both NPV and IRR. You won't find a careful definition for ROI, because there isn't one. If you're really curious to read more about this, you might want to track down a copy of "The Use of ROI in Information Security," an article that I wrote for the November 2008 issue of the ISSA Journal. This article describes this in a fair amount of detail.

It turns out that ROI is really just whatever argument will convince the decision-maker that you need to influence. Many ROI calculations start with NPV and tweak it to add things that are only relevant for security investments, but you don't have to do that. Any other calculation is just as valid. That's why there's really no need to define metrics like Return on Security Investment (ROSI) or security-adjusted ROI: because you can do those calculations and still call the result ROI.

The intent of ROI is to quantify the economic benefit of an investment, and if that means including measures of attacks or other losses that security technologies might reduce, that's fine. You can include elements like that in a calculation and still call the result ROI.

There are other good reasons to call a metric ROI instead of something else. In particular, other risk-management projects in an enterprise are typically justified by an ROI calculation, and if you're willing to allow a different metric for information security investments, it's only reasonable to expect other metrics to be used to justify other risk-management projects. That's probably a fight that's not worth having, so it's probably better to just keep the name ROI, even though it's really a metric that takes into consideration things that are particular to information security.

Doing that doesn't violate any accounting principle, because there's no careful definition of ROI that we really have to use. It's also in line with the intent of an ROI calculation, which is to measure the economic benefit of an investment. That's why I think that there's no need to develop a new metric called "security-adjusted ROI:" what you'll get is just ROI, but with a different name. You already have the freedom to include security benefits in ROI calculations. Why not just take advantage of this?

Comments

[Russell Thomas]

Yes, each organization may have it's own preferred method for ROI (a.k.a. finanical justification, more generally). But that doesn't mean that CISOs and security professionals are free to include or exclude anything they want as long as it helps justify their budget. That's a quick way to get laughed out of the CFOs office.

If you want to help your organization make better decisions and get better results in an arena as complicated as InfoSec, then you have to follow some rigorous method that have a sound theoretical basis, particularly in how it handles the many faces of risk and uncertainty.

>> "If you can estimate [HVAC], you should be able to estimate the ROI for information security purchases."

I would wager that estimating the full costs and benefits (direct + indirect, short-term + long-term) of InfoSec is *much* harder than estimating the costs/benefits of HVAC, which has very well understood performance models and easily-characterized statistical environment (weather patterns, load patterns, etc.). There are many articles that explore why InfoSec costs and benefits are hard to estimate (quantitatively), and any professional who has ever seriously tried to perform such analysis in a large organization can tell you their stories of woe and misery.

By the way, the sort of methods I'm recommending are not aimed at helping you justify Brand X firewall over Brand Y, and the like. From the perspective of enterprise risk management, those sorts of purchase decisions are in the noise. If that is what you care about, use simple ROI or TCO (total cost of ownership) and you are done.

If, instead, you are making major enterprise architecture decisions, insourcing vs. outsourcing, and web application management policies (i.e. major decisions), then you need a financial analysis/risk analysis framework that is more robust (and, yes, complicated).

Bottom-line: I'm agreeing with the people who think that so-called "security-enabled ROI" should be a research topic deserving both public and private funding, expanded to include the elements I have mentioned.

[Luther Martin]

I disagree that this adds confusion. ROI is actually a fairly simple comcept that people make much more complicated than it needs to be. The bottom line is that ROI is really whatever argument that you need to use to convince the decision-maker who needs to approve the budget for a security project. Because that's the case, you really need to tailor ROI argument to your target audience.

Some CIOs like to hear about cost savings. If that's the case, then emphasize them in your ROI calculation. If a different CIO likes to hear about a different fianancial metric, use that one instead. You can do this and still call your metric "ROI."

I think that it's too difficult to have a single methodology that makes sense in all cases, so that's probably the best you can do.

A related question that I also find interesting is how to infer the benefits from security investments. There are actually methodologies for estimating the ROI for other things that are fairly hard to understand, like the ROI for HVAC. If you can estimate things like that, you should be able to estimate the ROI for information security purchases.

[Russell Thomas]

I have published a "Total Cost of Security" method to solve these problems and provide a robust metric for financial investment and budget decision-making: http://meritology.com/resources/Total%20Cost%20of%20Cyber%20(In)security.ppt

One of the key elements of this method is to calculate costs and benefits in terms of a virtual "self-insurance fund" to cover the costs of low probability, high impact events. The recurring cost is reflected in the "self-insurance premium" for such a fund. (This method is analogous to methods used in financial analysis of derrivative securities, which also can't be evaluated using simple ROI or NPV methods alone. It's also compatible with Enterprise Risk Management methods.)

There are still some unsolved research problems and this method hasn't been tested empirically, but I contend it's a more promising approach than alternatives. Further research and development is needed, and I hope that the Summit report and recommendations encourage both government and private investment in such R&D projects.

Finally, InfoSec is *really* suffering due to lack of rational methods for evaluating and justifying investments and budgets. Following "best practices" is usually just "the blind leading the blind". It perpetuates myths and patterns of over-spending, under-spending, and mis-spending. Having some sort of evaluation method would be a "game-changer", as the Summit organizers were seeking.

(BTW, I wonder how many people at the Summit knew anything about these issues, or the underlying theory.)

[Russell Thomas]

Unfortunately, this post adds to confusion on the topic of ROI.

First, let's accept that "ROI" (return on investment) is mostly used as a generic label for any sort of financial justification for investment decisions. Therefore, the "security-adjusted ROI" mentioned at the Summit was probably NOT promoting a specific formula or method, but instead was appealing for some sort of financial justification method that adequately encompassed the business dynamics of InfoSec.

Second, there are a variety of definitions for ROI (return on investment, narrowly defined) that can be found in financial management texts. The short-coming of ROI, generally, is that it does not account for the time-value of money or the riskiness of the investment. That's where NPV and IRR come in. (NPV and IRR give nearly identical results in most cases.) ROI is OK for short time horizons but bad for long time horizons or for comparing investments with widely varying riskiness ("risky" in the financial sense).

Third, the debate between ROI (narrowly defined) vs. NPV vs. IRR is largely irrelevant for InfoSec. Anybody who puts this forward as the key issue is an idiot, IMHO. Yes, the details of methodology matter, but picking NPV over IRR and ROI won't get you any closer to a usable finanacial justification.

Fourth, this passage glosses over the real obsticles: "there's no need to develop a new metric... what you'll get is just ROI, but with a different name. You already have the freedom to include security benefits in ROI calculations."

The *real* obstacle to financial justification method for InfoSec investments is the difficulty of estimating and framing the "benefits" = reducing the likelihood and/or severity of InfoSec losses in the *future*, across the full range of risks. Another major challenge is appropriate accounting for all InfoSec costs (direct and indirect), not just "investments".
Furthermore, the uncertainty and "riskiness" of InfoSec investments does NOT fit the assumptions of NPV or IRR, where higher risk investments get a higher discount rate.

Therefore, it is no simple matter to just "include the security benefits in ROI calculations". It's like trying to use arithmetic alone to calculate the motion of a weight bouncing on the end of a spring. (It requires differential equations)