The classic quote from Warren Buffett about a financial crisis is also true for data centers. It's "only when the tide goes out do you discover who's been swimming naked," the multibillionaire businessman said.
Hurricane Sandy brought the tide in, literally. But the flooding in lower Manhattan had the same effect on data centers that, as Buffet observed, a recession has on financial services. It exposed the problems and shortcomings in the system.
There were data center outages. The causes range from flooded buildings that cut data centers off from fuel supplies, to backup generators that malfunctioned. The problems were extensive and the responses were often dramatic.
Zayo, which runs a data center in one of the most wired buildings in New York City, at 111 8th Ave., sent an "urgent communication" to customers Thursday. The temperature in one of its data center suites had reached 93 degrees Fahrenheit. Another communication soon followed, reporting that temps had "risen above 100 degrees."
Zayo conducted continuous conference calls for customers, detailed its multiple paths to a fix, including estimating how long it would take to get a 2MW backup generator delivered through New York traffic. The problem was a result of a failed isolation valve in the fuel system supplying a generator. It was fixed without any customer outages, and temperatures returned to 70 degrees.
In one case, at 75 Broad St., the response was just amazing.
Peer1, a data center that operates at 75 Broad St., was close to shutting down its data center after water flooded a basement, cutting off access to generator fuel supplies. That's when the employees of two customers of Peer1, Squarespace and Fog Creek Software, stepped in to carry fuel via a bucket brigade, up 17 flights.
There were many other reports of data center problems, particularly after the initial storm impact. Early in the week, Internap reported a data center outage at 111 8th Ave. due to a building-fed fuel system problem. Equinix reported a failed generator that "impacted service to several customers." The problems were eventually fixed at both sites.
New York's problems affected customers far and wide. Web sites were inaccessible and some networks were down.
TDS Telecommunications said it had experienced a "double failure" on two diverse routes of its network that go through New York. One source of trouble stemmed from a third-party data center outage in New York, and a second problem concerned an AT&T cable issue. About 9,000 data customers in New Hampshire were affected by the problem, since fixed.
"It's interesting," said DeAnne Boegli, TDS spokeswoman, "even when you have two options going into opposite sides of the city, they both get impacted."
Earlier in the week, monitoring firm Renesys, said the hurricane had caused outages in 10% of the networks in New York.
"Silencing ten percent of the networks in the New York area is like taking out an entire country the size of Austria, in terms of impact on the global routing table," wrote Renesys CTO James Cowie, on a blog.
It's too early to write a post mortem. The lower Manhattan data centers aren't expecting utility power until Sunday or Monday. Until then, the anxiety level will remain above normal, as data centers rely on generators, fuel trucks and ingenuity.
