And now, the end is near.
On 23 November 2012, Gerry Pennell will leave the job of a lifetime. Four years after signing up as CIO for the London 2012 Olympics and Paralympics, his IT team has won every technology gold medal going after successfully supporting the most connected Games ever. But even such a good thing must come to an end.
The technology hub on the 21st floor of London's Canary Wharf is full of empty desks and screens in boxes, as the IT team that peaked at 6,000 people during the Games moves on to other things.
"The office is quite empty, and perhaps a little sad, because so many members of the team have already left," says Pennell, in his first interview since the Olympics concluded.
"It's an interesting atmosphere in the office right now. The primary emotion is relief rather than euphoria, because a lot of things people predicted would go wrong, didn't – cyber attacks bringing systems down, mobile networks not allowing people to make phone calls, the systems not being ready on time. None of those things happened."
Now the Games are over, it's worth reflecting on the scale of the IT challenges that Pennell faced.
You come away with a sense of just how much can be achieved if you can only get everybody pointing in the right direction and working together
Gerry Pennell, London 2012 CIO
His team – consisting of in-house staff, volunteers and IT suppliers – deployed more than 110,000 pieces of equipment, mostly in the eight weeks before the opening ceremony, across about 30 venues, alongside 5,500km of cabling laid by BT. The team supported critical applications such as the Commentator Information System and results services, as well as monitoring 900 servers, 1,000 network and security devices and 9,500 PCs.
Much of the IT set-up was new to London – "An awful lot of new software," says Pennell – with 30% more results data processed than in Beijing in 2008, and for the first time, results being provided in real time for every Olympic sport on the web and to mobile devices.
"The last six to eight weeks before the opening ceremony was when the pressure was at its highest. That's when you're deploying all the physical equipment into the venues all at the same time," says Pennell.
"Getting a lot of stuff into a lot of venues simultaneously in a short amount of time was logistically challenging and most of the stress was around that – all aspects of it; people, technology, suppliers. It was about coordinating the activity of lots of organisation to get the right things to happen in the right sequence in venues, to get results in time and tested and commissioned before the Games started."
A typical day in the Olympics Technology Operations Centre
Every day during the Olympics and Paralympics, about 190 people at the Technology Operations Centre (TOC) monitored the IT systems, working on a three-shift pattern, 12 hours at a time. Pennell's day started early.
"My daily pattern was to come in at 6.30am for a morning call with the IOC [International Olympic Committee] about issues yesterday and what we're expecting today, and then after that brief our management on status," he says.
"Then depending on the day, I would typically spend some time in the TOC reviewing what was going on, such as cyber security or the volumes we were encountering on the website, etc. When I could, I would go out to the venues, not to interfere with what the troops were doing, but mainly to encourage them and check people were happy and focused and doing well."
Months of preparation and testing ensured there were no shocks, either internally or for the viewing public.
"We'd planned enough, there was tuning and tweaking going on, but we didn't have a problem. There were no moments of panic," says Pennell.
"When you're running a large technology operation you expect components to fail, and we had some component failures. The classic ones were equipment close to the field of play, in particular commentator terminals when it rained. During the Olympics we had quite a few sudden showers. In those, from time to time a PC would stop working, but we were ready for it and able to do a fast box-swap. We had those kinds of things going on," he says.
The most connected Olympics ever London2012.com became the most popular sports website in the world. It had 38.3 billion page views, peaking at 96,871 page views per second. Some 1.2 petabytes of data were transferred over the website, with a peak of rate of 22.8 Gbits/s. On the busiest day there were 13.1 million unique visitors. During the Games, the Olympic network which connects 94 locations (including 34 competition venues) carried 961TB of information. Olympic traffic to bbc.co.uk exceeded that for the entire BBC coverage of FIFA World Cup 2010 games. On the busiest day, the BBC delivered 2.8PB, with the peak traffic moment occurring when Bradley Wiggins won cycling gold and shifted 700Gbps. The BBC saw 12 million requests for video on mobile across the whole of the Games. During the Games, daily video traffic over BT's retail broadband network increased on average by 19%. Atos transmitted the results to the world's commentators in 0.3 seconds. Around 13.2 million minutes (or 220,000 hours) of BT Wi-Fi were used across Olympic Park venues. Acer provided 13,500 desktops, 2,900 notebooks, 950 servers and storage systems, and a number of tablet PCs. The mobile Olympics
If there was one surprise, it was the level of engagement from mobile phone users. London 2012 was the first Olympics of the smartphone era, and Pennell says he expected the consumerisation of IT to be a new challenge with little previous Games experience to fall back on. But the volumes of traffic generated reflected the scale of the shift in consumer behaviour.
"lt's the speed at which these things are changing, the BYOD [bring your own device] thing, the way the Games has to cope with that trend. It wasn't a surprise, but it came out very strongly in London," says Pennell.
"The most compelling statistic is that at peak moments over half the load on our web servers, which were hosted by BT, was coming from people either accessing the mobile site or using the mobile app. We expected this to be big in London, but I wouldn't have guessed it would be that big. At peak, on the website there were over half a million concurrent users, so it was pretty hot," he says.
"What was interesting was the interaction between those two things – the combination of live results available and people being able to get it on a mobile basis meant there was more volume of people looking for the latest updates than we had anticipated. We had to tune some things to make it work, which went fine. One of the highest peaks in the website use and mobile use in particular was the Andy Murray tennis final."
The pre-Games doomsayers warned that mobile networks wouldn't cope, but London 2012 organiser Locog had set up the Joint Olympic Operators Group (Joog) to work with the major mobile networks to ensure sufficient capacity.
"The infrastructure challenge was with the 3G networks in London, which were already stretched even without the Games," says Pennell. "[We had to consider how we were] going to provide a reasonable level of service to spectators for access to apps as well as for sharing user-generated content."
Through Joog, the mobile operators invested in a lot of shared infrastructure.
"We were offering a much better level of mobile networking experience at our venues than would be typical at a normal sporting event, so you could access everything on your device and share pictures and all the rest of it," he says.
"In a similar manner, we worked with BT across the Olympic Park. BT put in the world's largest high-density Wi-Fi system to enable services to be accessed from people's devices, and it all worked very well – my main measure being the absence of complaints. This was the Games when the consumerisation of technology really hit what the Olympics had to deliver."
Daily cyber attacks on Olympics systems
The other, less positive, expectation was that the Olympics would be an attractive target for hackers. This certainly proved to be the case. Pennell admitted to facing at least one particularly major cyber attack, and while he would not elaborate on the specifics, he says the security challenge was ongoing.
"We were attacked every day. Some of the attacks were fairly well orchestrated. Some just before the Games were automated," he says.
"We prepared for this well in advance so it didn't cause us any problems. It was interesting watching some of it, because some of the groups that were involved, from the hacktivist community, got fed up of trying to take us down and moved their attentions to other organisations.
"Entertainingly, some of this stuff was remarkably easy to track as there was the hashtag #letthegamesbegin, and they were publishing when they were going to attack – '7pm on Friday night let's have a go at the website'.
"There were other more serious attacks, but none of it caused a problem, but that's down to the right amount of planning and energy in preparation."
There is still work to do before Pennell leaves. His main focus is preparing for the formal de-brief in November to the organisers of the Rio 2016 Olympics in Brazil.
Meanwhile, all the equipment is being shipped back to a warehouse in Stevenage before being returned to suppliers or otherwise disposed of. Much of the hardware is being sold off at bargain prices to schools, part of a sustainability strategy that attracted praise from the IOC.
"The IOC commented we were the first technology delivery for a Games to have an integral sustainability strategy. That covered a lot of dimensions, from the energy efficiency of desktops to server virtualisation," he says.
"One of our responsibilities was to organise volunteers to run around the venues with bits of paper delivering results to VIPs and technical delegates. From Athens and Beijing, about 50 million sheets of paper were printed – that's a lot of paper. We cut that down to about 20 million.
"Plus we have a lot of data now in terms of carbon footprint to set a baseline for future games to refer back to. We're pretty proud of that."
The Olympics in the cloud
There is another important lesson Pennell hopes the IOC will learn – the potential for using the cloud to deliver the IT for future Olympics.
"The IOC understands my view that the Olympics is absolutely a textbook opportunity for a cloud implementation. It's peaky, with very high volume for very short period of time. From an economic point of view, you want a variable cost base, not a fixed cost base. The model for the Games, whereby you have to commission and build datacentres with vast rows of servers just for a short period of time, doesn't make economic sense," he says.
"The cloud and technology associated with cloud were not mature enough in 2008 for us to have been able to successfully manage that transition, but over the next few editions of the Games – and this is going to have to be driven by the IOC – a transition will have to take place."
Read more on the London 2012 Olympics IT Olympics media centre set to become datacentre Olympic Games 2012: Preparing for a mobile data deluge John Linwood, BBC CTO, on the Olympics output and digitisation CW500: The technology behind the BBC's Olympics coverage Skandia Team GBR analysed its way to victory at the Olympics Building an Olympics network Cisco to leave Olympic legacy in 25 London Network Academies Acer meets London 2012 Olympic challenge head-on More London 2012 Olympic Games Coverage
But London 2012 was the first to dabble in hosted services and cloud on a small scale.
"A number of our key services were not true cloud, but were hosted externally. Our IT incident management system was a hosted service, our arrivals and departures system was a hosted service, so we did more than previous Games to get to a place where we weren't doing everything in our datacentres," he says.
"In one incidence, the torch relay nomination process, that was based on a true public cloud offering and worked very well, and gave me six weeks of quite significant volumes at minimum cost. We used Microsoft Azure – it worked extremely well, and it was cheap."
Lessons learned from delivering IT for the London 2012 Olympics
So as Pennell starts to reflect, what are the lessons he has learned, and what could his CIO peers also take away from his Olympics experience?
"A lot of the lessons you learn from a project like this are no different from a big IT delivery in a commercial environment. The big one for me is always the same – if you can get the right team you can work miracles," he says.
"We were very fortunate in Locog, not just in technology, to get some pretty fantastic talent. Perhaps that's the thing the Olympics more than anything enables you to do. People really want to work on this project, which means you get the opportunity to hire some very good people. It's a privilege to lead that talented group."
But in briefing his successor, the technology director for Rio 2016, there will be valuable pointers that any CIO can benefit from.
"It's about getting the foundations correct in terms of contracts with providers, and in terms of clarity around scope, because one of the challenges in the Olympics is there's always more you can do. The trick is to work out what you're actually going to do and focus on those things. Make sure your budget is secured and you know what you can do. The fourth key foundation is building your team. If you can get those four things correct then regardless of whether it's the Olympics or any large technology delivery programme, you're in good shape," says Pennell.
At a personal level too, there is plenty for him to look back on.
You go into these things with varying degrees of terror
London 2012 CIO Gerry Pennell
"However much effort you put into testing, and into risk mitigation around cyber security, you never actually know it's going to work properly until you get there," he says.
"You go into these things with varying degrees of terror, especially when people say beforehand that the reputation of the UK technology industry is resting on you Gerry, and other helpful comments like that! You go into it with a nervous mindset. Then when you get to the point it's actually working, the results are being pumped out, scores of organisations are taking that results feed, supporting all the broadcasters, pumping out TV graphics etc. When you see that all working, it's pretty fantastic."
But there is also a definite sense from Pennell and all his team, as they move to new jobs too, of wanting to use their experience to prove to others – in the UK generally, as well as in UK IT – of what is possible if you put your mind to it.
"We started by talking about the general sense of nervousness from various quarters as to whether the UK could successfully deliver a huge technology programme on a massive integration, deployment and operational challenge such as this, with all the different providers involved. If there was one take-away I would want people to hear it would be this: Yes, of course, the UK absolutely has the capability to manage these types of projects well," says Pennell.
"You come away with a sense of just how much can be achieved if you can only get everybody pointing in the right direction and working together. The amount of stuff that's been done in London here is pretty phenomenal. You get a sense of what's possible, an aspirational sense of what you can achieve if you get everybody focused on it."
What next for ex-London 2012 CIO Gerry Pennell?
So if this was the job of a lifetime, how do you follow that? Pennell says he has the luxury of taking his time, and picking the best opportunity. He has already turned down a number of approaches.
"At this point I have no plans. I have some thoughts but not plans. I'm in no rush. I want to do some sailing when I leave. Then when the right thing comes along I'll be in a position to take it," he says.
"I won't try to follow this. I'll probably do something slightly different. You'll have to wait and see – sorry to be a tease. I genuinely don't know yet. You can be certain it needs to be quite challenging, otherwise I'm not going to be interested. In many ways, running a project in this context is no different from in a corporate context – until you get to the Games itself, when it is entirely and completely different."
But the one thing Pennell does know is exactly where he plans to be at the moment his Olympics CIO experience finally comes to an end. "My official end date is 23 November. I get made redundant on that day. My plan is to be made redundant while sitting on Copacabana beach in Rio."
