Four days of non-stop manual efforts by datacentre staff at Peer 1 Hosting to deliver gallons of fuel to 17th floor generators has helped the colocation provider's Manhattan facility remain operational despite hurricane Sandy.
Superstorm Sandy brought a destructive storm surge to New York City on Monday 29 October, flooding many areas in Lower Manhattan and cutting off electricity in many parts of the city.
Peer 1 Hosting's Manhattan colocation facility was affected by the electricity outage.
Power and network connections are two major components required to keep a datacentre functional, said Dominick Monkhouse, EMEA managing director and senior vice-president of customer service at Peer 1.
As the network runs on waterproof fibre optic cable, the Peer 1 datacentre did not face network outages. But an electricity outage meant it had to switch on its power generators to light and cool the facility and keep the IT equipment running.
Drawing datacentre power from back-up generators
But it was not easy. Normally, during power outages, the facility uses power from generators in the basement, but a two-metre water surge in the building meant the basement generators were flooded (pictured), making them redundant.
Read more on datacentre power requirements Global census shows datacentre power demand grew 63% in 2012 Google to use wind energy to power Oklahoma datacentre Microsoft's datacentre energy waste highlights datacentre bad practices 'Big three' datacentre power suppliers hold nearly 50% of the market Google opens €75m energy-efficient datacentre facility in Dublin Goldman Sachs adopts modular datacentres to save costs, improve efficiency
"This meant we had to use our back-up generators located on the 17th floor," said Monkhouse.
The generators burn between 35 and 40 gallons of diesel every hour, so the datacentre managers decided to use the generators to power only the equipment and lighting.
"For datacentre infrastructure cooling, we decided to use free-air cooling and kept the windows open," he said.
But it could not rely on free-air cooling for long, because the hot air outlet of another datacentre from two floors below was just below its window, making the centre warm.
The team then decided to run the more critical zone of its datacentre – which serves 30 customers – with the back-up generators. It also decided to keep the temperature of the datacentre slightly higher than usual.
"Normally we operate the facility at 23°C, but to conserve fuel in the generators we ran it at a higher temperature of around 26-27°C," said Monkhouse.
But as the fuel ran down, the staff had to top up the diesel to keep the generators powered. As part of its disaster recovery strategy, Peer 1 has an emergency fuel supply contract, so it had fuel pump trucks arrive onsite to power the back-up generators, but the hosepipes were not long enough to reach the 17th floor.
A group of 30 people carried gallons of diesel up 17 floors to power the generators to keep the datacentre online
The datacentre staff, local people and two of its customers formed a group of 30 people to carry gallons of diesel up 17 floors to power the generators to keep the datacentre online. Together, the team carried 455 gallons of fuel for the generators between Monday and Thursday.
But on Thursday, the company received calls expressing labour union objection, which meant it could no longer use the team of 30 people to add fuel to its generators manually. It also received a warning from the coastguard to evacuate.
"We have managed to secure long hosepipes to reach the 17th floor, but we don't have pumps," he said. "The truck is stuck in New Jersey and does not have enough fuel to drive down to 75 Broad Street in Manhattan.
"We have fuel to last us for another eight hours," said Monkhouse on Thursday, "but we are hoping the truck will reach us within eight hours so the efforts of the staff do not go waste."
Providing business continuity in the face of disaster
Datacentre managers from Peer 1's Toronto facility are arriving at the Manhattan location with tools and spare parts to re-provision the datacentre infrastructure, he added.
Read more on business continuity Case study: Disaster recovery boosts business continuity Business continuity must become a boardroom issue Evaluating business continuity options on cloud services Plan for IT disruptions to continue business as usual Business continuity strategy: Employees are the vital link
The company has also created an online forum – Peer 1 Forum – to provide customers with live updates.
"Currently we are reviewing all options and bringing in resources from out of state to deliver fuel to the generators. We will keep you posted on an hourly basis via email and online forums of the status of generator power and provide you as much lead time as possible if it becomes necessary for you proceed with a graceful shutdown of your servers in order to preserve your data," the update read.
Carl Brooks, analyst, IT infrastructure and cloud at Tier 1 and 451 Research, said: "I'd say well done in the face of unavoidable catastrophe, and the positive that will come out of this is that everyone involved is going feel very fit, and next time around, there may well be a way to cope with this exact dilemma.
"There are finite limits on every contingency – these guys (and every other datacentre and carrier hotel in New York and New Jersey, by the way) made plans to enter emergency mode, arranged for special fuel deliveries, guarded their operations as well as they could, and a once-in-a-lifetime event took out a heap of those plans at a swoop."
