There is a new and somewhat unexpected threat to the growth and adoption of cloud computing services: performance and availability issues. Identified in a recently released study by Australian researchers, stress tests of Amazon, Google, and Microsoft's respective cloud offerings have apparently revealed dramatically intermittent service issues.

IT News Australia has the article.

This is surprising since one of the primary advantages to using cloud services should be better price/performance and price/availability metrics than most companies could reliably achieve internally. Economies of scale are supposed to allow cloud providers to build world-class infrastructure and spread the costs sufficiently among subscribers that all share better service at less cost. This isn't simply a marketing trope; it's an economic reality that governs many of the normal market interactions we all perform from day to day, from shopping at the local grocery store to flipping on a light switch.

Cloud service providers, particularly Amazon, which is the only one of the three to have taken the "beta" label off of most of its services, have been emphasizing this heavily in their own marketing efforts, highlighting very small businesses which have successfully used their product to rapidly scale offerings at a fraction of the cost and time that doing so internally would have required. Using EC2 to accommodate a sudden extra influx of demand is one of its selling points.

So the finding that response times varied by a factor of twenty depending on the time of day services are being accessed is something of a shock. The massive absorption of overload is being called into question.

Regrettably, few details of the study are available. While the "factor of twenty" sounds threatening, if the highest time is still blindingly fast, there might not be much to it. There is also little mention of the methodology or precise nature of the tests involved. Most CIOs understand that your picture of services can vary dramatically based on the metrics chosen and the methods used to obtain them. Dry technical tests may not reveal useful pictures of the functionality available to the end-user.

Then again, two of the three services tested are still in limited beta testing; you would expect considerable variability as development continues, and in fact the article acknowledges as much, stating, "But researchers also found that the three platforms delivered wildly variable performance results as Amazon, Google and Microsoft trialled, added and dropped new features." The whole point of new features is to affect performance; without a corresponding judgment of the value-added effect of the feature, performance fluctuations alone don't substantiate the overall judgement delivered by the article.

Nonetheless, it's encouraging that someone is taking the first steps at digging in and quantifying and comparing performance of cloud services. As mentioned in previous posts, I think that the level of mystery surrounding all these offerings are the biggest obstacle to adequately judging their utility, and the more investigation that is being done, the sooner those mysteries can be dispelled. As researcher Anna Liu says, "None of the platforms have the kind of monitoring required to have a reasonable conversation about performance." Until that changes, the enterprise can only proceed slowly in adopting any of them.