Kansas Health Institute | Information For Policy Makers. Health for Kansans.

TOPEKA — Kansas Department of Health and Environment Secretary Roderick Bremby held a press conference today to explain the cause of a major computer crash at the agency as it entered its 12th day and to describe the ongoing work of technicians and vendors to restore vast quantities of data while whittling away some new backlogs of information submitted since the systems went down.

"This outage has caused many problems for Kansans and we are very sorry for the inconvenience this has caused," Bremby said. "Now, I can say with certainty that KDHE is on the road to recovery."

Bremby said the state's online immunization registry and several other key databases were working again as of Monday but that it still remained uncertain when all of the agency's 24.7 terabytes of affected data would be fully restored and the agency and its various clients would be back to normal.

Bremby said the problems were caused by an Aug. 5 hardware failure that then caused a software problem that damaged the agency's storage area network or SAN. Additional problems to the SAN were detected Aug. 11 and a new one was delivered and installed on Aug. 12. Since then agency technicians and vendors have been working "around the clock," to restore the agency's data.

The problems never compromised the security or integrity of the agency's data, Bremby said, but they "couldn't have occurred at a worst time."

The agency's immunization registry crashed just as many parents around the state were taking their children to local health departments for back-to-school vaccinations. That prompted some local health officials to turn away families that didn't come prepared with paper records of their immunizations.

"As a parent, I understand the need for documents to enroll your child in school and the frustration with not being able to attain them," Bremby said. "This has been hard for everyone and we appreciate your patience."

State IT officials have called the crash perhaps the biggest and longest in state history. One official told KHI News Services it was, a "perfect storm."

Bremby at the press conference called it, "a never event," in that it combined a hardware failure with a software glitch that neither agency IT experts nor vendors could have or had foreseen.

But he said he believed there were steps the agency and others could take to avoid future occurrences of, "the cascading effect," that resulted when the device failure apparently triggered the software problems that prompted shutting down the systems.

He said those problems were compounded by the fact that the agency relies heavily on "virtual servers" instead of using more costly hardware or box servers. Instead of the "never event" knocking out a single server and perhaps a couple of databases it affected 150 servers and 61 of the agency's 107 databases, including email, immunization records, and other health informatics including the records for births, deaths, marriages and divorces.

Agency computer technicians have been working nearly non-stop since Aug. 5, Bremby said. Cots had been set up in the KDHE data rooms.

Further complicating the problems for an already exhausted staff, he said, was the death of a son of a key KDHE IT technician during the course of the computer crash. Bremby said the employee left work for a short while Monday for the funeral but was planning to return once the services were over.

"It's that kind of dedication," Bremby said after the press conference, his voice catching with emotion.

Bremby said total cost of dealing with the problem, "could be north of $600,000 to $700,000."

He said some of that expense could be the responsibility of a vendor, though he declined to identify the company, saying he had not yet, "talked with corporate," about that.

He praised a second vendor, On Track, for working closely with the agency to begin restoring backup data soon after the initial problems were discovered.

Bremby said the agency began relying heavily upon the virtual servers about a year ago and had not experienced problems with them prior to the Aug. 5 crash.