When seawater flooded streets in lower Manhattan Monday night, it filled the bottom floors of the basement of an office building at 75 Broad St. and its lobby up to four feet. This was not a good development for the data center operated by Peer1 Hosting.
Customers and others helped lug fuel up 17 floors to keep the rooftop generator at 75 Broad St. in New York City running and a data center in the building humming. (Photo: Squarespace)
Peer1's data center had smoothly switched over to generator power, but when Con Edison cut electric power to lower Manhattan, the rooftop generator couldn't access the 20,000 gallon fuel tank in the flooded basement. Its pumping system was disabled by storm waters from Hurricane Sandy.
The rooftop generator was using as much as 40 gallons of fuel an hour to run the nearly 13,000-square-foot data center, and the "day tank" had limited capacity. Time was running out.
Early Tuesday, Peer1's Broad Street customers were warned that fuel supplies would soon be exhausted. Among those who got the email was Anthony Casalena, the founder and CEO of Squarespace, a Web publishing and content management firm. He was in his powerless apartment in downtown Manhattan.
Casalena started Squarespace in 2003 with $30,000 borrowed from his father. Today, his company has 102 employees and hundreds of thousands of customers. As a private company, it doesn't give out exact numbers.
Casalena has been using the Peer1 data center since his business started. He was familiar with its operation and may now be its largest customer. He left his downtown apartment to see for himself what was happening, to offer help, and hopefully, he said, to delay any shut down "until the last possible second." His customers were warned and he continued to report on the status of the data center.
Casalena said it quickly became clear that the data center operations team didn't have a good way to measure the fuel supplies in the rooftop day tank. "They don't really know how much fuel was left," he said.
Near the time the data center was supposed to shut down, Casalena said the data center team determined that there was a half tank left. That triggered a new plan.
Casalena said there were some 55-gallon oil drums onsite. The data center manager suggested trying to get fuel supplies to the roof. They made arrangements with one of the fuel trucks on the street for supply. Fuel trucks have become a common sight since the power outage.
Casalena was among those helping to carry fuel to the roof.
The generator-fueling effort got organized. More people begin showing up from Squarespace and another large user, Fog Creek Software. They included software developers, systems engineers, sales representatives, executives and support personnel.
A network of people, stationed at every staircase landing, formed a bucket brigade passing fuel from one person to the next. Several dozen may have been involved over multiple shifts, including some day laborers who were also hired to help.
"Over the next night into the morning, we were able to successfully continue to coordinate fuel trucks coming and manually move hundreds of gallons of diesel to the roof and keep the thing online," Casalena said.
There were moments when they felt they weren't moving fast enough. At one point, a generator light came on showing low fuel.
"It's kind of just very dramatic," said Casalena. But they managed thanks to a "huge group effort," he said.
"It looks like we have operationalized this to the point where we can make it work -- I can't honestly believe it," said Casalena, "It somehow worked."
They now have enough fuel on the roof to stagger deliveries.
"Morale was super high," said Joel Spolsky, the CEO and co-founder of Fog Creek Software, who runs a number of businesses out of the Peer1 data center. "People were just absolutely itching to get there and help and do something and make some kind of contribution."
"It was hard, but people were kind of excited to have a mission to work on," said Spolsky, who is also co-founder of StackExchange, a major site for developers. That operation was moved to a data center in Oregon, but Spolsky has other work in the Peer1 center that can't be easily moved.
Spolsky, who tweeted about the efforts, said he even had offers from customers and people onTwitter to help out. "I think people feel kind of helpless sitting at home watching the news, and they were so excited to be able to do something even if it was hard to make a substantial contribution," he said.
The efforts continued to restore the basement pumping system. Water kept finding its way into the basement, but Casalena said that as of early Wednesday evening, it appeared the basement flooding situation was getting under control.
Both Casalena and Spolsky praised the Peer1 New York operations staff for their efforts. Peer1, which has 19 data centers worldwide, has extra help arriving tonight, driving up from a Virginia because of problems getting flights to New York.
Along with the efforts to get the building's pumping system in operation, Robert Miggins, senior vice president of business development at Peer1, said the company is working on alternative methods to pump fuel up to the roof.
Miggins said the data center remains online, in part, because the company didn't keep the situation it faced in New York a secret. "We choose to tell customers all the details and that is what led them to show up and enable them to lend a hand," he said.
Without that notice, "we wouldn't have had the manpower there to actually bring the fuel up in time," Miggins said. "There's a lot of good will, and there's a lot of hard work and there's a few lucky bounces for good measure," he said.
No one is going to feel comfortable until power is fully restored. Spolsky said his staff has been marking the company's physical systems to speed reassembly at another data center if the need arises.
If there's any takeaway from this experience, it may be this: Casalena said he's glad he went to the data center when he did to help and to take an active role. Otherwise, "I don't know how this would have played out," he said.
