Some Perspective on System Redundancy

Posted on November 16, 2011


Hospital Operations and Facilities leadership: I have some ammunition for your next internal discussions on system redundancy—or at least an outside perspective. It comes in the fashion of a glimpse inside credit card company, Visa. In this month’s issue of Fast Company, writer Michael Fitzgerald takes a rare look inside the brains of Visa’s network headquarters, which handles its 8,000 transactions a second. If your hospital ever debates how much of your operations “really needs” 100% emergency power or how much and how many days (hours—you’re kidding, right?) your hospital can run on non-grid utilities, you may want to read this article.

To briefly summarize, Visa built the Operations Center East (OCE) as its newest data center to handle the data security of its 150 million daily transactions. Once inside (the outdoor campus and interior security functions have many redundancies, of course), its level of design for uninterrupted operations in the face of ‘what-ifs’ is stunning.  Some of the highlights:

  • Every major system in the data center—mainframes, air conditioners, batteries—has a backup
  • Facility is designed to withstand earthquake and tornado disruptions
  • In the loss of power, batteries kick in.  Diesel generators back up the batteries, and the generators have a backup.
  • In the loss of water, a 1.5 million gallon water tank will serve.  Three wells back up the water tank.
  • Four conduits serve electricity—just in case one is accidentally taken out during construction (sound familiar?)
  • Every pod (the facility has at least five) has two rooms with uninterruptible power supplies to handle power surges
  • Each pod has two rooms with 1,000 heavy-duty batteries to create a make-shift “world’s largest laptop computer” for 30 minutes
  • Each pod has two diesel generators with 4 megawatt capacity—yes, they are soundproofed.
  • The OCE has five, 1250-ton chillers that chemically treat the water to prevent corrosion

Note well:  these are infrastructure investments a public company has made simply to keep its operations going, which happens to be millions of financial transactions a day. Visa has deemed what it does at this particular facility so important that the redundancy is worth the peace-of-mind that comes with knowing practically every ugly scenario that could take down its systems, man-made and natural, has been designed for.  And like the Apple moniker ‘there’s an app for that’, Visa can say:  “yeah, there’s a backup for that”.

Easy for me to say, but at the end of the day all Visa does is help money change hands. Sure, they facilitate commerce but lives are not at stake—not like at a hospital. In light of recent natural disasters like Hurricane Katrina and the Joplin tornado, which completely destroyed entire hospitals, hospitals should strongly consider their role in the community and what it means to be provide care. Sometimes knowing your facility is the only thing that will have lights on after a Category 5 or 8.0, and having your community know that, is the greatest care you can provide.