The Ohio Department of Public Safety successfully moved its mainframe applications to a Windows-based system, and all the work was done in-house.
The decision to keep the work in-house prompted much debate and skepticism, but there was no ducking the question that the agency had to retire the mainframe technology that had long supported its most important operations.
The project began in 2007, and the switchover was completed in March of this year. The mainframe is now unplugged.
The mainframe had supported legacy code in applications that dated back to the 1980s.
There were about 2,000 programs running on the mainframe, about 50 of which no one knew anything about. As engineers retired, knowledge about many of the applications running on the mainframe was lost.
At the beginning, all of the migration options were daunting.
Rewriting everything to Microsoft's .Net programming language would be too expensive and would take too long, but staying on a mainframe was too expensive as well.
Bringing in consultants to handle a transition to Windows might have cost as much as $10 million, said Keith Albert, the chief of IT governance and strategic direction for the department.
Therefore, the department trained its staff of long-time mainframe veterans who had worked mostly in IBM Pacbase code. The migration involved .Net training and a move from a hierarchical to a relational database.
The agency spent some $250,000 on training and hired a few contractors to augment the staff.
The core of the development team included about 30 IT staff members who, by Albert's count, spent about 84,500 person-hours on the project.
There was some skepticism about the project at the start.
Albert said IT officials found little information about successful migrations, and that was disconcerting. "We couldn't find anything out there that said we were going down the right path," he said.
The lack of information made an impression on Albert, who wrote a paper detailing the steps taken during the agency's migration and the issues the development team and management dealt with. The 18-page paper was completed last month and was posted Thursday on the department's website.
The document is a compelling read, with details about the problem, the options, the internal debate and the lessons learned along the way.
The title of the paper -- Exodus Project - Pigs Really Do Fly! -- is a remark made by a staff member who, early on, thought the project was impossible.
The agency's Unisys ClearPath Dorado mainframe had operated dependably for many years and was running more than 2,000 programs. However, the model the agency owned had reached its end of life and it had to be replaced by this year.
Albert said he would have stayed on the mainframe for several more years if Unisys had extended the product's life and cut its MIPS charges, which were running about $1 million a year. The agency paid an additional $200,000 in maintenance fees.
The high MIPS (millions of instructions per second) charges and the fact that the hardware was reaching the end of the line were grating for Albert.
He compared the situation to "owning a car that runs fine, but there is only one place you can buy gas for the car and that gas station refuses to sell any more gas for the car after that X date."
"There was no reason why we couldn't use the mainframe another five years," said Albert. The latest machine had been installed in 2001 but was actively maintained by adding fiber cards, I/O processors and I/O channels, as needed.
But there were other motivations for migrating off the mainframe. One was its Pacbase code. Working with it was becoming increasingly difficult as veteran staffers retired. "There are very few Pacbase programmers in the world anymore," said Albert.
There was skepticism internally about whether the legacy code could run as reliably on a Windows platform as it did on the mainframe -- if it could run at all. But with about 30% of the IT staff eligible for retirement in three years and 50% eligible in five years, a migration was all but inevitable.
The agency will still need Pacbase skills for years to come, but the long-term plan is to rewrite the 2 million lines of legacy code for a .Net environment. "As long as my Pacbase programmers stick around long enough to keep things running, my .Net programmers can rewrite it," he said.
Albert's paper outlines some of the struggles faced by the development team.
There "was no expert judgment available or models to follow," wrote Albert. The initial work could have been completed more quickly "had the decision to re-platform not been repeatedly debated."
Albert also made note of the "valleys of despair" the team faced along the way, and he included a list of lessons learned.
Despite its challenges, the team had executive sponsorship throughout the project.
Users were also engaged. "The users were more confident with the change because the re-platforming approach meant the same code was executing, and that meant very few errors occurred," wrote Albert.
There were issues related to the code. There was no inventory of 5% of the code, and there were about 50 programs running that no one knew anything about. The people responsible for them had long since left the organization.
Some of the issues were overcome by using a Fujistu compiler that worked with Pacbase and allowed the legacy code to execute in a Windows environment.
The department sought bids for x86 systems. It selected Unisys ES7600 servers.
If Unisys had backed off on the need for MIPS charges, Albert said "we would have a much different conversation."
MIPS charges have been a longstanding approach in the mainframe market, according to Charles King, an analyst at Pund-IT. As onerous as the charges may seem, "the vast majority of mainframe owners don't seem to have a problem it," said King.
Vendors argue that the mainframe is such a unique technology "that it deserves a unique kind of sales, service and maintenance model," said King.
Unisys, in a written response to a query about Ohio's migration, said the department "made its decision to transition off of its ClearPath system before Unisys introduced the new NextGen platforms and modernization technology."
"Evolving the ClearPath line to a common Intel-based architecture has enabled Unisys to improve price/performance options for our clients and reduce maintenance costs to levels comparable to those for other Intel-based platforms," Unisys said.
The mainframe and the Windows servers ran in parallel for three and a half months before the switch became permanent.
In the end, Albert estimates that the project will save the agency between $7 million and $10 million over the next five years.
