RSSROI metrics for SOA
I posted a couple of weeks ago a reference to a recent Gartner study showing that some 40% of companies engaging in SOA projects do not associate ROI metrics with those projects. I posited at the time that perhaps this wasn't quite the travesty that it seemed, or that other commenters held it to be, since the benefits of a functioning SOA are mostly downstream and secondary, and can be difficult to assess without a correspondingly larger measurement effort which may will cost more than the benefits could accrue.
Today Joe McKendrick comes to the rescue with a list of metrics that Gartner proposes to evaluate SOA project success.
1. Improved efficiency, particularly with respect to business processes execution
2. Lower process administrative costs
3. Higher visibility on existing/running business processes
4. Reduced number of manual, paper-based steps
5. Better service-level effectiveness
6. Quicker implementation of processes
7. Quicker time to market
8. Shorter (overall) project cycles
9. Overall reduction in the total cost of application development and maintenance
Those metrics all seem reasonable to me; but accurately establishing them as the result of the architecture changes is another matter, which gets back to my original point. Are you going to attribute all your decrease in process administration costs to the SOA implementation? How do you single out what SOA contributes versus other efforts? Or do you shut down every other cost-saving effort in these difficult times just so you can see how the SOA is doing? Is SOA the only thing affecting your application development and maintenance costs? Can you control for all the other factors? How do you really know that any of these things are coming about as a result of your SOA effort?
I suppose the real point is that you have to look at the general conditions and take some things on faith (if your costs go down consistently after you implement SOA, there must be a relationship, right?), but I'm not sure if that really counts as rigorous ROI measurement… or if it is in fact much different from simply launching into the process with no more than a gut feeling. To me, ROI metrics should be solid, or you shouldn't waste the time on them. You need them exactly when the effects are too fuzzy to ballpark or guesstimate; when they themselves are simply guesstimates of their own, I'm not sure I see the value.
If you're going to wing it, at least have the courage of your convictions… don't come up with funny numbers to support the effort.
Well if you follow that rule Scott, you may never get any ROI metric at all - ROI metrics in most business situations will be a bit fuzzy since it is always a fluid situation with many variables.
We cannot take a “bean counter” approach and say if it has estimates it is no good. I gave an example of project that posted a solid ROI on Joe McKendrick’s original article - the point being the objective of an ROI is to ascertain whether you should invest in the technology / architecture or later to determine if the investment was actually worth it.
So if you can “quantify” some of the benefits and they add up to a solid ROI you can stop there and dont need to sweat out more details.
The simple test is “Does it add to my bottom line ?” - atleast that is what the CFO/CIO approach would be !
Well if you follow that rule Scott, you may never get any ROI metric at all - ROI metrics in most business situations will be a bit fuzzy since it is always a fluid situation with many variables.
We cannot take a “bean counter” approach and say that if it has estimates it is no good. I gave an example of project that posted a solid ROI on Joe McKendrick’s original article - the point being the objective of an ROI is to ascertain whether you should invest in the technology / architecture or later to determine if the investment was actually worth it.
So if you can “quantify” some of the benefits and they add up to a solid ROI you can stop there and dont need to sweat out more details.
The simple test is “Does it add to my bottom line ?” - atleast that is what the CFO/CIO approach would be !
I’m not sure we’re actually saying different things here; my point is that this isn’t really “ROI” and we’re being deceptive in calling it that. I’m not saying hard ROI is necessary to launch or justify these projects and I think the soft ROI measurement you are suggesting is possible is even more fuzzy with SOA than in many situations by dint of the long-term, downstream nature of its effects… it’s a bit like trying to predict the benefits of theoretical hard science research.
So if that’s the case, let’s just accept it, and stop trying to win over the CFO by pretending we can measure this to that extent. I like your idea of attempting to quantify only some of the benefits, and that might be an approach with merit, but don’t pretend it is ROI in the conventional sense of the term.