The improbable story of a New York City data center saved by a fuel bucket brigade is not told with fondness, even if there is a sense of pride and humor in its retelling.
Those involved recall the stink of diesel fuel in dark stairwells illuminated by flashlights. There was the sheer physical exhaustion of carrying buckets of diesel up 17 floors to deposit them into a rooftop generator's fuel tank. There was the sleeping on floors and the lack of showers, and the unrelenting stress of knowing that their efforts could suddenly unravel and the data center could fail.
The Peer 1 bucket brigade carries diesel to the rooftop generator's fuel tank.
And of course all involved had to live with the broader weight brought on by concern about the impact that Hurricane Sandy was having on their friends and relatives, and on New York itself.
The effort to save the data center during and after the October 2012 storm is captured in a documentary-style film that Peer 1, the data center owner, produced and showed Tuesday to a group of its customers and employees.
The film, featuring photos of data center rescue efforts and interviews with participants, doesn't attempt to romanticize the events or add a false sense triumph to the actions of those involved. It just lays them out, leaving viewers with a sense of sheer exhaustion as they watch the efforts of the participants and share their sense of uncertainty about the outcome.
Peer 1 posted the video on YouTube Wednesday night.
The concept of carrying fuel to the roof "seemed like a ridiculous idea," said Michael Pryor, president of Fog Creek Software, a large user of the Peer 1 data center, in an interview. "It just didn't seem feasible."
Pryor's idea, retold during the film, was to bring some extra pumps from a fish tank in his office to the nearby Peer 1 data center to see if they might help get fuel to the rooftop generator. During the screening, that suggestion drew healthy laughter from the audience, and Pryor himself now chuckles at the idea.
But the fact that he actually thought of using fish tank pumps helps convey the sense of urgency that the data center workers and customers experienced in the early going as they struggled to come up with ideas for keeping a rooftop generator running as it burned 40 gallons of fuel an hour out of a 200-gallon tank.
Fog Creek makes a software management platform used by developers, and the company and its customers would have suffered if the 5,000 square-foot data center at 75 Broad St. in lower Manhattan went down.
Michael Mazzei, Peer 1 data center manager. (Photo: Patrick Thibodeau/Computerworld)
Peer 1 is a short walk from Fog Creek's offices, and as the software company grew so did its use of the data center. The data center provides Fog Creek's network and infrastructure for customer equipment.
There's an area set aside for Peer 1 customers to work on their systems.
Data center employee Jeffery Burns recalled that he heard the sound of a waterfall in the elevator shaft as floodwaters entered the lobby and eventually flooded the basement and disabled the fuel pumping systems.
"Everybody in Manhattan has the same problem, how to relocate infrastructure that -- for the last 50 years -- has safely lived in basements," said Michael Mazzei, manager of the Peer 1 Broad Street data center.
Despite the damage caused by Hurricane Sandy, data centers won't be leaving Manhattan anytime soon. The island is one of the most networked places in the world.
The Peer 1 data center is in the former International Telephone & Telegraph headquarters building and has multiple network connections that give customers quick access and low latency. It's also a short walk to the waterfront.
New York Mayor Michael Bloomberg this month announced a multibillion-dollar, multidecade plan to protect the city from powerful storms, and from the impact of climate change. The plan includes the construction of barriers in low-lying parts of the city.
Bloomberg's plan has broad IT and telecommunications implications.
It seeks a higher level of reliability and service for telecommunication facilities. The specific recommendations include requiring new hospitals "to increase their IT and telecommunications resiliency by installing two independent points-of-entry for telecom and communication to reduce the risk of outages from a single supplier."
For Fog Creek, and likely many other businesses, the most significant long-term impact of Hurricane Sandy may stem from the fact that it caught the company without a backup plan. Fog Creek has been working to fix that weakness in the months since.
For instance, the software company has improved its data replication capabilities and is developing an ability to switch operations to a separate facility in an emergency. One tool now runs in Amazon's cloud service, though that migration had been planned prior to the storm.
The storm made clear to Fog Creek that it hadn't paid enough attention to the likelihood that something bad, like a massive storm, could ever happen, said Pryor. That was a positive outcome of the experience, he said, "because it did kick us in the butt and get us to fix a lot of things that were broken."
In his office, Mazzei points to the part of the floor he slept on -- next to his phone.
The bucket brigade wasn't the group's first move. Initially, they tried hauling 55-gallon drums up the 17 flights of stairs, but the physical exertion proved too much.
Gradually, though, a realistic plan came into place. The team acquired pumps, hoses, buckets and other supplies from Home Depot and other places. Organization, rules and efficiency followed.
One rule: "Go slowly, work safely, always have your flashlight," said Mazzei.
For 72 hours, the effort continued and the data center was able to remain in operation until a more stable fuel supply was established. For the future, one option the building owners are considering is installing submersible pumps that could operate during a flood, said Mazzie.
There were a lot of motivations at work in manning the bucket brigade. For customers, it was about keeping their businesses up. For the data center workers, it was about not failing.
No one is really taking credit for saving the data center.
Mazzei is very matter-of-fact about the effort, and describes it more as an organic process that developed on its own as ideas and processes emerged collectively.
Burns, a film editor who works part time at the data center, said the team rallied around Mazzei. "It was just really important to not let him down," he recalled, "not because he was going to get upset, but because he never let us down."
This article, "A Data Center Story for the Ages: The Fuel Bucket Brigade," originally appeared on Computerworld.com.
