RSSCan a single data center provide a cloud?
You've probably already had enough of the dreary definitions debate that has been happening around the sidelines of the new utility computing movements, where dreadfully pedantic pundits spar over the precise meaning of terms like "cloud", "SOA", or the various aaSes. Too bad, I say; here is more pedantry, coming at you!
You're probably familiar with hosting provider Rackspace, whose mantra of "Fanatical Support" is a clarion call to beleaguered CIOs who are fed up with the lackluster approach to support that many outsourcing and hosting companies adopt to keep costs down. Of late, Rackspace has been jumping hard to get up on the cloud bandwagon, offering up "cloud servers" from $10 a month and courting the Internet cognoscenti (the featured recommendation showing right now on their Rackspacecloud website is not from a Fortune 500 company, or a major industry publication, but rather from an "influential blogger") with the service.
Yesterday's tempest in a teapot outage at the Rackspace data center in Texas demonstrated the backlash that can come from courting influential bloggers and then failing to keep up with their expectations. But it also revealed something interesting about the service itself, which is that it all appears to be run out of the single datacenter in DFW.
Now, I had not mentioned multiple, redundant hosting locations in my own personal definition of what constitutes cloud computing, and from the perspective which I approached it, I'm not sure that any of the physical components of provisioning should really matter. Yet I think in the back of my mind I sort of assumed some sort of geographic redundancy. I'm not so foolish as to think there are no circumstances in which cloud services might not fail despite that feature, but I guess it just seemed to me that, at a bare minimum, your cloud wouldn't wink out because of a peering facility network issue at a single data center.
So, is my definition wrong? Or should "cloud" suggest something more robust than it's namesake?
Scott you correctly identified the need for superior infrastructure for cloud computing. There have been too many instances in the last six months of outages. (T mobile, Rackspace, Twitter, Google, Google G-mail, Blackberry, Amazon, Equinix and others). Without pointing a finger on a specific problem, let’s raise a flag at a few very specific solutions. Redundant hosting locations and constant evaluations of the facilities infrastructure will reduce outages.
I never use the term eliminate because there will always be human intervention with the equipment. Evaluating the infrastructure will involve electrical, cooling, rack redundancy and personnel training. I had an experience at a large data center this summer where the facility was rated as a Tier 4 data center with 96hrs of generator backup. However when the site experienced a utility outage the data center went dark after an ATS failed to transfer to generator. There was an ATS bypass breaker installed but no one on staff had been trained how to use it in case of an emergency.
After 45 minutes the UPS batteries were depleted and half of the electrical buss was lost. Maintenance personnel attempted to cross connect the two busses from the switchgear but was never trained on how to do it. Instead of cross connecting the switchgear the input feeder was opened and the facility went dark.
Time for CIO’s to take the steps to properly evaluate their infrastructure or have an independent agency evaluate it for them. Just as a fire departments run drills, so should data centers. Run outage drills, operate circuit breaks and test the knowledge of their staff. Just a few basics for “How not to build your house on a sand foundation.”
I completely agree. I’m never certain whether or not to be surprised that CIOs aren’t more proactive in these matters. On the one hand, there are plenty of cautionary tales such as the on you related, and the role of the CIO has been primarily a defensive one to date, so you would imagine that ferreting out and guarding against these issues would be more on the radar. On the other hand, many IT departments essentially operate in a perpetual crisis mode and never get ahead of the curve of the current disaster to prevent future calamities.
I think there has been, in executive and other circles, a sort of aura imbued around the term “data center” that somehow makes them seem invulnerable and reliable, when in both your experience and mine, that hasn’t proven to be the case. Many data center operators aren’t much more proficient than the corporate IT departments that are outsourcing to them; it’s just hidden behind the mystique and the whole outsourcing wall, and CIOs are as susceptible to marketing drivel as anyone else. But one of the primary predictors of potential problems, unsurprisingly, is complexity; and data centers have become tremendously complex. Your story illustrates exactly that point. It’s the same sort of layers and layers of untested complexity that has led to so many of the industrial disasters of our era. Unless we can learn that lesson, the same sort of problems are destined to follow us into the era of digital utilities.